Commit 1b336f74 authored by Marin Jankovski's avatar Marin Jankovski

Merge branch 'upgrade_omnibus-version' into gitlab_omnibus

parents 7a3bd3f5 0abab93b
Omnibus CHANGELOG
=================
v4.1.0 (Unreleased)
### Bug Fixes
- Config.append_timestamp is now properly handled by the build_version
DSL. For some users, this may introduce a change in behavior. To
revert to the old behavior set append_timestamp to false in
`omnibus.rb` or use --override append_timestamp:false at the command
line.
v4.0.0 (December 15, 2014)
--------------------------
......
![Omnibus Icon](lib/omnibus/assets/README-logo.png) Omnibus
===========================================================
[![Gem Version](http://img.shields.io/gem/v/omnibus.svg)][gem]
[![Build Status](http://img.shields.io/travis/opscode/omnibus.svg)][travis]
[![Build Status](http://img.shields.io/travis/chef/omnibus.svg)][travis]
[gem]: https://rubygems.org/gems/omnibus
[travis]: http://travis-ci.org/opscode/omnibus
[travis]: http://travis-ci.org/chef/omnibus
Easily create full-stack installers for your project across a variety of platforms.
......@@ -133,6 +133,11 @@ Some DSL methods available include:
| `package` | Invoke a packager-specific DSL |
| `compress` | Invoke a compressor-specific DSL |
By default a timestamp is appended to the build_version. You can turn
this behavior off by setting `append_timestamp` to `false` in your
configuration file or using `--override append_timestamp:false` at the
command line.
For more information, please see the [`Project` documentation](http://rubydoc.info/github/opscode/omnibus/Omnibus/Project).
### Software
......@@ -219,7 +224,7 @@ end
Since the software definitions are simply ruby code, you can conditionally execute anything by wrapping it with pure Ruby that tests for the version number.
#### Sharing software definitions
The easiest way to share organization-wide software is via bundler and Rubygems. For an example software repository, look at Chef's [omnibus-software](https://github.com/opscode/omnibus-software). For more information, please see the [Rubygems documentation](http://guides.rubygems.org/publishing/).
The easiest way to share organization-wide software is via bundler and Rubygems. For an example software repository, look at Chef's [omnibus-software](https://github.com/chef/omnibus-software). For more information, please see the [Rubygems documentation](http://guides.rubygems.org/publishing/).
It is recommended you use bundler to pull down these gems (as bundler also permits pulling software directly from GitHub):
......@@ -253,6 +258,40 @@ $PWD/config/software/foo.rb
The first instance of `foo.rb` that is encountered will be used. Please note that **local** (vendored) softare definitions take precedence!
Version Manifest
----------------
Git-based software definitions may specify branches as their
default_version. In this case, the exact git revision to use will be
determined at build-time unless a project override (see below) or
external version manifest is used. To generate a version manifest use
the `omnibus manifest` command:
```
omnibus manifest PROJECT -l warn
```
This will output a JSON-formatted manifest containing the resolved
version of every software definition.
Changelog
---------
STATUS: *EXPERIMENTAL*
`omnibus changelog generate` will generate a changelog for an omnibus
project. This command currently assumes:
- version-manifest.json is checked into the project root
- the project is a git repository
- each version is tagged with a SemVer compliant annotated tag
- Any git-based sources are checked out at ../COMPONENT_NAME
- Any commit message line prepended with ChangeLog-Entry: should be
added to the changelog.
These assumptions *will* change as we determine what works best for a
number of our projects.
Caveats
-------
......
......@@ -67,8 +67,33 @@ You can inspect the cache keys like this:
git --git-dir=$CACHE_PATH tag -l
You can manually remove a cache entry using `git tag --delete
$TAG`. In theory, you can share the cache among build slaves using
`git clone/fetch`.
$TAG`.
## Build Slaves ##
You can share the cache among build slaves using `git clone/fetch`.
Using git bundles is another option (if you're using Jenkins, the bundle
can be persisted as an artifact):
+ Backup: `git --git-dir=$CACHE_PATH bundle create $CACHE_BUNDLE --tags`
+ Restore: `git clone --mirror $CACHE_BUNDLE $CACHE_PATH`
(CACHE_BUNDLE is anywhere you're persisting the bundle to)
### Important Caveat ###
Note that Omnibus caches instructions *exactly as they will be executed*
(e.g. Omnibus caches `make -j 9`, it doesn't cache `make -j #{workers}`).
So, if you intend to share your build cache among slaves, you should ensure
that their build environments are identical (failing which you'll bust the
cache with commands that depend on the environment).
## Mac OS X Caveats ##
When running a build vm on OS X, note that you will likely run into
trouble if you attempt to locate your cache on your local
......
......@@ -65,7 +65,7 @@ Some DSL methods available include:
| `vendor` | The name of the package producer |
| `license` | The default license for the package |
| `priority` | The priority for the package |
| `catetory` | The catetory for this package |
| `category` | The category for this package |
If you are unfamilar with any of these terms, you should just accept the defaults. For more information on the purpose of any of these configuration options, please see the RPM spec.
......
......@@ -18,6 +18,7 @@ Given(/^I have an omnibus project named "(.+)"$/) do |name|
write_file('omnibus.rb', <<-EOH.gsub(/^ {4}/, ''))
# Build configuration
append_timestamp false
cache_dir './local/omnibus/cache'
git_cache_dir './local/omnibus/cache/git_cache'
source_dir './local/omnibus/src'
......
......@@ -72,10 +72,20 @@ module Omnibus
autoload :NullPublisher, 'omnibus/publishers/null_publisher'
autoload :S3Publisher, 'omnibus/publishers/s3_publisher'
autoload :Manifest, 'omnibus/manifest'
autoload :ManifestEntry, 'omnibus/manifest_entry'
autoload :ManifestDiff, 'omnibus/manifest_diff'
autoload :ChangeLog, 'omnibus/changelog'
autoload :GitRepository, 'omnibus/git_repository'
autoload :SemanticVersion, 'omnibus/semantic_version'
module Command
autoload :Base, 'omnibus/cli/base'
autoload :Cache, 'omnibus/cli/cache'
autoload :Publish, 'omnibus/cli/publish'
autoload :ChangeLog, 'omnibus/cli/changelog'
end
class << self
......
......@@ -46,6 +46,10 @@ module Omnibus
def semver
new.semver
end
def build_start_time
new.build_start_time
end
end
# Create a new BuildVersion
......@@ -102,7 +106,7 @@ module Omnibus
#
# format: YYYYMMDDHHMMSS example: 20130131123345
if Config.append_timestamp
build_version_items << build_start_time.strftime(TIMESTAMP_FORMAT)
build_version_items << build_start_time
end
# We'll append the git describe information unless we are sitting right
......@@ -120,6 +124,26 @@ module Omnibus
build_tag
end
# We'll attempt to retrive the timestamp from the Jenkin's set BUILD_ID
# environment variable. This will ensure platform specfic packages for the
# same build will share the same timestamp.
def build_start_time
@build_start_time ||= begin
if ENV['BUILD_ID']
begin
Time.strptime(ENV['BUILD_ID'], '%Y-%m-%d_%H-%M-%S')
rescue ArgumentError
error_message = 'BUILD_ID environment variable '
error_message << 'should be in YYYY-MM-DD_hh-mm-ss '
error_message << 'format.'
raise ArgumentError, error_message
end
else
Time.now.utc
end
end.strftime(TIMESTAMP_FORMAT)
end
# Generates a version string by running
# {https://www.kernel.org/pub/software/scm/git/docs/git-describe.html
# git describe} in the root of the Omnibus project.
......@@ -179,9 +203,9 @@ module Omnibus
# @return [nil] if no pre-release tag was found
def prerelease_tag
prerelease_regex = if commits_since_tag > 0
/^\d+\.\d+\.\d+(?:-|\.)([0-9A-Za-z.-]+)-\d+-g[0-9a-f]+$/
/^v?\d+\.\d+\.\d+(?:-|\.)([0-9A-Za-z.-]+)-\d+-g[0-9a-f]+$/
else
/^\d+\.\d+\.\d+(?:-|\.)([0-9A-Za-z.-]+)$/
/^v?\d+\.\d+\.\d+(?:-|\.)([0-9A-Za-z.-]+)$/
end
match = prerelease_regex.match(git_describe)
match ? match[1] : nil
......@@ -237,26 +261,6 @@ module Omnibus
private
# We'll attempt to retrive the timestamp from the Jenkin's set BUILD_ID
# environment variable. This will ensure platform specfic packages for the
# same build will share the same timestamp.
def build_start_time
@build_start_time ||= begin
if ENV['BUILD_ID']
begin
Time.strptime(ENV['BUILD_ID'], '%Y-%m-%d_%H-%M-%S')
rescue ArgumentError
error_message = 'BUILD_ID environment variable '
error_message << 'should be in YYYY-MM-DD_hh-mm-ss '
error_message << 'format.'
raise ArgumentError, error_message
end
else
Time.now.utc
end
end
end
# Pulls out the major, minor, and patch components from the output
# of {#git_describe}.
#
......@@ -267,8 +271,13 @@ module Omnibus
#
# @todo Compute this once and store the result in an instance variable
def version_composition
version_regexp = /^(\d+)\.(\d+)\.(\d+)/
version_regexp.match(git_describe)[1..3]
version_regexp = /^v?(\d+)\.(\d+)\.(\d+)/
if match = version_regexp.match(git_describe)
match[1..3]
else
raise "Invalid semver tag `#{git_describe}'!"
end
end
end
end
......@@ -14,6 +14,8 @@
# limitations under the License.
#
require 'time'
module Omnibus
class BuildVersionDSL
include Logging
......@@ -33,7 +35,7 @@ module Omnibus
@output_method = nil
if version_string
@build_version = version_string
self.build_version = version_string
elsif block_given?
instance_eval(&block)
construct_build_version unless from_dependency?
......@@ -78,7 +80,7 @@ module Omnibus
# @return [String]
def explain
if build_version
"Build Version: #{@build_version}"
"Build Version: #{build_version}"
else
if from_dependency?
"Build Version will be determined from software '#{version_dependency}'"
......@@ -105,6 +107,43 @@ module Omnibus
source_options[:from_dependency]
end
def build_version=(new_version)
@build_version = maybe_append_timestamp(new_version)
end
# Append the build_start_time to the given string if
# Config.append_timestamp is true
#
# @param [String] version
# @return [String]
def maybe_append_timestamp(version)
if Config.append_timestamp && !has_timestamp?(version)
[version, Omnibus::BuildVersion.build_start_time].join("+")
else
version
end
end
# Returns true if a given version string Looks like it was already
# created with a function that added a timestamp. The goal of this
# is to avoid breaking all of the people who are currently using
# BuildVersion.semver to create dates.
#
# @param [String] version
# @return [Boolean]
def has_timestamp?(version)
_ver, build_info = version.split('+')
return false if build_info.nil?
build_info.split('.').any? do |part|
begin
Time.strptime(part, Omnibus::BuildVersion::TIMESTAMP_FORMAT)
true
rescue ArgumentError
false
end
end
end
# Determines the build_version based on source_type, output_method.
#
# @param version_source [Omnibus::Software] Software object from which the
......@@ -120,10 +159,10 @@ module Omnibus
end
output = output_method || :semver
@build_version = version.send(output)
self.build_version = version.send(output)
when :version
if version_source
@build_version = version_source.version
self.build_version = version_source.version
else
raise "Please tell me the source to get the version from"
end
......
......@@ -580,7 +580,8 @@ module Omnibus
#
def build
log.info(log_key) { 'Starting build' }
shasum # ensure shashum is calculated before build since the build can alter the shasum
log.internal(log_key) { 'Cached builder checksum before build: #{shasum}'}
if software.overridden?
log.info(log_key) do
"Version overridden from #{software.default_version} to "\
......
require 'omnibus/git_repository'
module Omnibus
class ChangeLog
CHANGELOG_TAG = "ChangeLog-Entry"
attr_reader :end_ref
def initialize(start_ref=nil, end_ref="HEAD", git_repo=GitRepository.new('./'))
@start_ref = start_ref
@end_ref = end_ref
@git_repo = git_repo
end
def authors
git_repo.authors(start_ref, end_ref)
end
def changelog_entries
entries = []
current_entry = []
git_repo.commit_messages(start_ref, end_ref).each do |l|
if blank?(l)
entries << current_entry
current_entry = []
elsif tagged?(l)
entries << current_entry
current_entry = Array(l.sub(/^#{CHANGELOG_TAG}:[\s]*/, ""))
elsif !current_entry.empty?
current_entry << l
end
end
entries << current_entry
entries.reject(&:empty?).map(&:join)
end
def start_ref
@start_ref ||= git_repo.latest_tag
end
private
attr_reader :git_repo
def blank?(line)
line =~ /^[\s]*$/
end
def tagged?(line)
line =~ /^#{CHANGELOG_TAG}:/
end
end
end
module Omnibus
class ChangeLogPrinter
def initialize(changelog, diff, source_path="../")
@changelog = changelog
@diff = diff
@source_path = source_path
end
def print(new_version)
puts "## #{new_version} (#{Time.now.strftime('%Y-%m-%d')})"
print_changelog
if !diff.empty?
print_components
puts ""
end
print_contributors
end
private
attr_reader :changelog, :diff, :source_path
def print_changelog(cl=changelog, indent=0)
cl.changelog_entries.each do |entry|
puts "#{' ' * indent}* #{entry.sub("\n", "\n #{' ' * indent}")}\n"
end
end
def print_components
puts "### Components\n"
print_new_components
print_updated_components
print_removed_components
end
def print_new_components
return if diff.added.empty?
puts "New Components"
diff.added.each do |entry|
puts "* #{entry[:name]} (#{entry[:new_version]})"
end
puts ""
end
def print_updated_components
return if diff.updated.empty?
puts "Updated Components"
diff.updated.each do |entry|
puts sprintf("* %s (%.8s -> %.8s)",
entry[:name], entry[:old_version], entry[:new_version])
repo_path = ::File.join(source_path, entry[:name])
if entry[:source_type] == 'git' && ::File.directory?("#{repo_path}/.git")
cl = ChangeLog.new(entry[:old_version], entry[:new_version], GitRepository.new("#{repo_path}"))
print_changelog(cl, 2)
end
end
puts ""
end
def print_removed_components
return if diff.removed.empty?
puts "Removed Components"
diff.removed.each do |entry|
puts "* #{entry[:name]} (#{entry[:old_version]})"
end
puts ""
end
def print_contributors
puts "### Contributors\n"
changelog.authors.each do |author|
puts "* #{author}"
end
end
end
end
......@@ -61,12 +61,47 @@ module Omnibus
#
# $ omnibus build chefdk
#
method_option :output_manifest,
desc: "Create version-manifest.json in current directory at the end of the build",
type: :boolean,
default: true
method_option :use_manifest,
desc: "Use the given manifest when downloading software sources.",
type: :string,
default: nil
desc 'build PROJECT', 'Build the given Omnibus project'
def build(name)
project = Project.load(name)
manifest = if @options[:use_manifest]
Omnibus::Manifest.from_file(@options[:use_manifest])
else
nil
end
project = Project.load(name, manifest)
say("Building #{project.name} #{project.build_version}...")
project.build_me
project.download
project.build
if @options[:output_manifest]
FileUtils.mkdir_p('pkg')
File.open(::File.join('pkg', 'version-manifest.json'), 'w') do |f|
f.write(JSON.pretty_generate(project.built_manifest.to_hash))
end
end
end
register(Command::ChangeLog, 'changelog', 'changelog [COMMAND]', 'Create and view changelogs')
CLI.tasks['changelog'].options = Command::ChangeLog.class_options
#
# Generate a version manifest for the given project definition
#
# $ omnibus manifest PROJECT
#
desc 'manifest PROJECT', 'Print a manifest for the given Omnibus project'
def manifest(name)
puts JSON.pretty_generate(Project.load(name).built_manifest.to_hash)
end
#
......
#
# Copyright 2015 Chef Software, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
require 'omnibus/changelog'
require 'omnibus/changelog_printer'
require 'omnibus/manifest_diff'
require 'omnibus/semantic_version'
module Omnibus
class Command::ChangeLog < Command::Base
namespace :changelog
#
# Generate a Changelog
#
# $ omnibus changelog generate
#
method_option :source_path,
desc: "Path to local checkout of git dependencies",
type: :string,
default: "../"
method_option :starting_manifest,
desc: "Path to version-manifest from the last version (we attempt to pull it from the git history if not given)",
type: :string
method_option :ending_manifest,
desc: "Path to the version-manifest from the current version",
type: :string,
default: "version-manifest.json"
method_option :skip_components,
desc: "Don't include component changes in the changelog",
type: :boolean,
default: false
method_option :major,
desc: "Bump the major version",
type: :boolean,
default: false
method_option :minor,
desc: "Bump the minor version",
type: :boolean,
default: true
method_option :patch,
desc: "Bump the patch version",
type: :boolean,
default: false
method_option :version,
desc: "Explicit version for this changelog",
type: :string
desc 'generate', 'Generate a changelog for a new release'
def generate
g = GitRepository.new
if @options[:skip_components]
diff = Omnibus::EmptyManifestDiff.new
else
old_manifest = if @options[:starting_manifest]
Omnibus::Manifest.from_file(@options[:starting_manifest])
else
Omnibus::Manifest.from_hash(JSON.parse(g.file_at_revision("version-manifest.json",
g.latest_tag)))
end
new_manifest = Omnibus::Manifest.from_file(@options[:ending_manifest])
diff = Omnibus::ManifestDiff.new(old_manifest, new_manifest)
end
new_version = if @options[:version]
@options[:version]
elsif @options[:patch]
Omnibus::SemanticVersion.new(g.latest_tag).next_patch.to_s
elsif @options[:minor] && !@options[:major] # minor is the default so it will always be true
Omnibus::SemanticVersion.new(g.latest_tag).next_minor.to_s
elsif @options[:major]
Omnibus::SemanticVersion.new(g.latest_tag).next_major.to_s
end
Omnibus::ChangeLogPrinter.new(ChangeLog.new(),
diff,
@options[:source_path]).print(new_version)
end
end
end
......@@ -432,10 +432,16 @@ module Omnibus
['omnibus-software']
end
# The solaris compiler to use
# Solaris linker mapfile to use, if needed
# see http://docs.oracle.com/cd/E23824_01/html/819-0690/chapter5-1.html
# Path is relative to the 'files' directory in your omnibus project
#
# For example:
#
# /PATH/files/my_map_file
#
# @return [String, nil]
default(:solaris_compiler, nil)
default(:solaris_linker_mapfile, "files/mapfiles/solaris")
# --------------------------------------------------
# @!endgroup
......@@ -488,6 +494,11 @@ module Omnibus
# @return [Integer]
default(:fetcher_read_timeout, 60)
# The number of retries before marking a download as failed
#
# @return [Integer]
default(:fetcher_retries, 5)
# --------------------------------------------------
# @!endgroup
#
......
......@@ -16,9 +16,14 @@
require 'openssl'
require 'pathname'
require 'omnibus/logging'
module Omnibus
module Digestable