Commit a20b6cad authored by Marin Jankovski's avatar Marin Jankovski

Merge branch 'upgrade_omnibus_version' into 'gitlab_omnibus'

omnibus version to v5.0.0



See merge request !2
parents 1b336f74 fd95fde3
......@@ -83,6 +83,14 @@ Gemfile.lock
*.tmproject
tmtags
# avoid generic ctags files too
tags
gems.tags
gemtags
.tags
.gem.tags
.gemtags
.tags_sorted_by_file
### SublimeText ###
# workspace files are user-specific
......
rvm:
- 1.9.3
- 2.0.0
- 2.1
bundler_args: --jobs 7 --without docs local
- 2.2
bundler_args: "--jobs 7 --without docs local"
branches:
only:
- 2.0-stable
- 3.0-stable
- master
script: bundle exec rake travis:ci
notifications:
hipchat:
slack:
on_change: true
on_failure: true
on_success: false
on_pull_requests: false
rooms:
# Build Statuses
- secure: JdLOITSPHW5xk8eoFOmH1Js5PT6iFgswUG8fIqNq69ie2Qws2K58hPDR6HW2NBEsBb7dW1S5jT6V9RHhm0ykekvEJVa5AjebO3EsZa+Cu/VahFMg4DL+SGYgKlKrZQosd+EgVQQ3C9gj0dApGtKqf2Ej7RNESwCPY8SOduTC6d8=
# Release Engineering
- secure: PmOaqp2DFyy79VpCuvSYILblK6tP0eDa9fDG9X2j+wggoosBkmlGfnSxp3A2TXlUyK62mr6/B89dRkL2aDWTJ/gqaZ34Elx2rV5S2A5YJIx9stA2+4iXeSlOF2YMNKGwBVmF/kp/nNoG2FqUkUBrRurWAOXZnwxKIhQ7+kLVb/0=
secure: udLCoBl71xJBsW3P+YqZbA5xfaB8l4IIF6SKEXpjIUYNYyP6m77uoY9PZ/M/5EN1ojPXDxsZ6SiPEJifVpkX5Hqd0eu1MwWyRXvOwiDf/nkg2Y3mqwUQRb3LDYWZxPn+xp5mHePe3mvTICR1zH49yx7fdZs8Qvzhh3Lno/w/qZ4=
Omnibus CHANGELOG
=================
v4.1.0 (Unreleased)
v5.0.0 (November 10, 2015)
--------------------------
### New Features
- Wind River Linux 5 support for Cisco Nexus devices (#539)
- [artifactory publisher] Support custom properties in Artifactory publishing (#568)
- [msi packager] New "fast" mode for MSI packager (#565)
- Change the `appbundler` DSL method to not make an apps dir
- Unit and functional tests now run on Windows (and are tested by Appveyor) (#556, #557)
### Bug Fixes
- [msi packager] Fix missing package name in signature (#541)
- [rpm packager] Fix building RPMs on ARM platforms (#542)
- [bff packager] Fix regression with AIX package ownership in staging directory (#553)
- [solaris packager] Use the proper architecture value in Solaris packages (#554, #555)
- Add info message for publish cli corner case (#551)
- [net fetcher] missing checksum raises exception (#549)
### Potentially Breaking Changes
- Dropped Ruby 1.9.x support (#567)
v4.1.0 (September 1, 2015)
-------------------------
### New Features
- Allow semver prefixes that start with "v"
- Reset target_shasum in PathFetcher after every fetch, preventing erroneous GitCache misses for builds that produced local artifacts during the previous omnibus run.
- Allow users to specify options to the underlying FileSyncer in a Software definition
- Add CPPFLAGS in with_standard_compiler_flags
- Add a Fedora dist tag to package name and spec release
- Copy distribution.xml.erb also when creating project with --pkg-assets
- Manage version manifests with Omnibus. Omnibus now creates a version manifest in text and JSON.
- Retry failed downloads
- Support inclusion of email address in Wix template
- Allow solaris to use mapfiles
- A new `omnibus generate changelog` command generates an opinionated CHANGELOG entry based on metadata in commit messages.
- Add warnings on empty globs.
- Raspberry Pi 2 support
- Clone git repositories with --recursive flag
- Add ability to sign MSIs
- Support running Omnibus with 64-bit ruby on Windows
- Create an Artifactory build for published packages
- Add a "windows_arch" omnibus option to choose 32/64 bit builds.
- Set directory ownership/permissions to match the filesystem package.
- Allow Windows 10 builders
- Allow omnibus to build "bundled" Windows installers
- Set perms on control files per Ubuntu Software Center's lintian checks.
- Replace uber-s3 dependency with aws-sdk
### Bug Fixes
- Config.append_timestamp is now properly handled by the build_version
DSL. For some users, this may introduce a change in behavior. To
revert to the old behavior set append_timestamp to false in
`omnibus.rb` or use --override append_timestamp:false at the command
line.
- Do not memoize current_revision to avoid returning incorrect data. Memoizing current_revision was previously causing version_for_cache to return the pre-fetch revision of the software, leading us to erroneously restore an older revision from the cache.
- Clean up "files listed twice" warnings. This should allow signing RPMs on EL 7
- dpkg uses arch name ppc64el (instead of ppc64le) for little endian
- Override equality operator for ManifestEntry. Prior to this, we simply tested basic object identity, which would result in many "updated" dependencies
- Correctly determine git repository when running in a subdirectory
- Forcibly remove a non-empty project dir before cloning because if multiple projects are using the same maching for building, they might have two different software definitions that share the same name but a different source.
- Accomidate hardlinks properly
- When building spec file, handle path names that contain spaces
- Add libkvm and libprocstat to freebsd whitelist
- Properly render ERB templates that do not have variables
- Change git fetcher to correctly fetch when repo name is the same but branch/version is different.
### Potentially Breaking Changes
- `Omnibus::SemanticVersion.new` now raises `Omnibus::InvalidVersion` instead of `Omnibus::SemanticVersion::InvalidVersion`
- Cache Builder#shasum before Builder#build to ensure consistent result.
- Update chef-sugar to 3.0 which adds ppc64le support.
- The MSI packager now adds the architecture to the msi name. The file names go from `package_name-build_version-build_iteration` to `package_name-build_version-build_iteration-arch`.
v4.0.0 (December 15, 2014)
--------------------------
......
# Maintainers
This file lists how the Omnibus project is maintained. When making changes
to the system, this file tells you who needs to review your patch - you need a
simple majority of maintainers for the relevant subsystems to provide a :+1: on
your pull request. Additionally, you need to not receive a veto from a
Lieutenant or the Project Lead.
Check out
[How Chef is Maintained](https://github.com/chef/chef-rfc/blob/master/rfc030-maintenance-policy.md#how-the-project-is-maintained)
for details on the process, how to become a maintainer, lieutenant, or the
project lead.
## Project Lead
* [Seth Chisamore](https://github.com/schisamo)
## Maintainers
* [Daniel DeLeo](https://github.com/danielsdeleo)
* [Jay Mundrawala](https://github.com/jaym)
* [Lamont Granquist](https://github.com/lamont-granquist)
* [Scott Hain](https://github.com/scotthain)
* [Seth Vargo](http://github.com/sethvargo)
* [Steven Danna](https://github.com/stevendanna)
* [Yvonne Lam](http://github.com/yzl)
* [Thom May](https://github.com/thommay)
......@@ -24,7 +24,7 @@ Prerequisites
-------------
Omnibus is designed to run with a minimal set of prerequisites. You will need the following:
- Ruby 1.9+
- Ruby 2.0.0+
- Bundler
......@@ -204,7 +204,7 @@ Additionally, there are a number of DSL methods avaiable inside the `build` bloc
For more DSL methods, please consult the [`Builder` documentation](http://rubydoc.info/github/opscode/omnibus/Omnibus/Builder).
You can support building multiple verisons of the same software in the same software definition file using the `version` method and giving a block:
You can support building multiple versions of the same software in the same software definition file using the `version` method and giving a block:
```ruby
name "ruby"
......@@ -274,6 +274,24 @@ omnibus manifest PROJECT -l warn
This will output a JSON-formatted manifest containing the resolved
version of every software definition.
Whitelisting Libraries
----------------------
Sometimes a platform has libraries that need to be whitelisted so the healthcheck
can pass. The whitelist found in the [healthcheck](https://github.com/chef/omnibus/blob/master/lib/omnibus/health_check.rb)
code comprises the minimal required for successful builds on supported platforms.
To add your own whitelisted library, simply add the a regex to your software
definition in your omnibus project as follows:
```
whitelist_file /libpcrecpp\.so\..+/
```
It is typically a good idea to add a conditional to whitelist based on the specific
platform that requires it.
*Warning: You should only add libraries to the whitelist that are guaranteed to
be on the system you install to; if a library comes from a non-default package
you should instead build it into the package.*
Changelog
---------
......
version: "{build}"
os: Windows Server 2012
platform:
- x64
environment:
matrix:
- ruby_version: "200"
clone_folder: c:\projects\omnibus
clone_depth: 1
skip_tags: true
branches:
only:
- master
install:
- SET PATH=C:\Ruby%ruby_version%\bin;C:\Ruby21%ruby_version:~3%\Devkit\mingw\bin;%PATH%
- echo %PATH%
- ruby --version
- gem --version
- gem install bundler --quiet --no-ri --no-rdoc
- bundler --version
- cp C:\Ruby21%ruby_version:~3%\Devkit\mingw\bin\bsdtar.exe C:\Ruby21%ruby_version:~3%\Devkit\mingw\bin\tar.exe
- appveyor DownloadFile http://curl.haxx.se/ca/cacert.pem -FileName C:\cacert.pem
- set SSL_CERT_FILE=C:\cacert.pem
build_script:
- bundle install
test_script:
- bundle exec rake unit
- bundle exec rake functional
- bundle exec rake acceptance
......@@ -22,7 +22,7 @@ repository) is
customize the location of the cache in the `omnibus.rb` config file
using the key `git_cache_dir`. For example:
git_cache_dir "/opt/ominbus-caches"
git_cache_dir "/opt/omnibus-caches"
## How It Works ##
......
Feature: omnibus publish
Scenario: Overriding publishing platform
* I run `omnibus publish artifactory fake * --platform debian`
Scenario: Providing platform mappings file
* I have a platform mappings file named "platform_mappings.json"
* I run `omnibus publish artifactory fake * --platform-mappings platform_mappings.json`
* the output should contain:
"""
Publishing platform has been overriden to 'debian'
"""
Scenario: Overriding publishing platform version
* I run `omnibus publish artifactory fake * --platform-version 7`
* the output should contain:
"""
Publishing platform version has been overriden to '7'
Publishing will be performed using provided platform mappings.
"""
require 'aruba/api'
Given(/^I have an omnibus project named "(.+)"$/) do |name|
create_dir(name)
create_directory(name)
cd(name)
write_file("config/projects/#{name}.rb", <<-EOH.gsub(/^ {4}/, ''))
......@@ -27,3 +27,15 @@ Given(/^I have an omnibus project named "(.+)"$/) do |name|
package_tmp './local/omnibus/pkg-tmp'
EOH
end
Given(/^I have a platform mappings file named "(.+)"$/) do |name|
write_file(name, <<-EOH.gsub(/^ {4}/, ''))
{
"ubuntu-10.04": [
"ubuntu-10.04",
"ubuntu-12.04",
"ubuntu-14.04"
]
}
EOH
end
......@@ -4,10 +4,12 @@ require 'aruba/in_process'
require 'omnibus/cli'
Aruba.configure do |config|
config.command_launcher = :in_process
config.main_class = Omnibus::CLI::Runner
end
Before do
# Reset anything that might have been cached in the Omnibus project
Omnibus.reset!(true)
Aruba::InProcess.main_class = Omnibus::CLI::Runner
Aruba.process = Aruba::InProcess
end
......@@ -276,27 +276,12 @@ module Omnibus
def appbundle(app_name, options = {})
build_commands << BuildCommand.new("appbundle `#{app_name}'") do
bin_dir = "#{install_dir}/bin"
embedded_apps_root = "#{install_dir}/embedded/apps"
embedded_app_dir = "#{embedded_apps_root}/#{app_name}"
gemfile_lock = "#{embedded_app_dir}/Gemfile.lock"
appbundler_bin = windows_safe_path("#{install_dir}/embedded/bin/appbundler")
# Ensure the main bin dir exists
FileUtils.mkdir_p(bin_dir)
# Ensure the embedded app directory exists
FileUtils.mkdir_p(embedded_apps_root)
# Copy the application code into place
FileUtils.cp_r("#{Omnibus::Config.source_dir}/#{app_name}", embedded_apps_root)
# Delete any top-level `.git` directory
FileUtils.rm_rf("#{embedded_app_dir}/.git")
# Prepare the environment
options[:env] ||= {}
env = with_embedded_path || {}
env["BUNDLE_GEMFILE"] = gemfile_lock
options[:env].merge!(env)
shellout!("#{appbundler_bin} '#{embedded_app_dir}' '#{bin_dir}'", options)
shellout!("#{appbundler_bin} '#{Omnibus::Config.source_dir}/#{app_name}' '#{bin_dir}'", options)
end
end
expose :appbundle
......@@ -488,10 +473,16 @@ module Omnibus
# @return (see #command)
#
def copy(source, destination, options = {})
build_commands << BuildCommand.new("copy `#{source}' to `#{destination}'") do
command = "copy `#{source}' to `#{destination}'"
build_commands << BuildCommand.new(command) do
Dir.chdir(software.project_dir) do
FileSyncer.glob(source).each do |file|
FileUtils.cp_r(file, destination, options)
files = FileSyncer.glob(source)
if files.empty?
log.warn(log_key) {"no matched files for glob #{command}"}
else
files.each do |file|
FileUtils.cp_r(file, destination, options)
end
end
end
end
......@@ -499,7 +490,7 @@ module Omnibus
expose :copy
#
# Copy the given source to the destination. This method accepts a single
# Move the given source to the destination. This method accepts a single
# file or a file pattern to match
#
# @param [String] source
......@@ -511,10 +502,16 @@ module Omnibus
# @return (see #command)
#
def move(source, destination, options = {})
build_commands << BuildCommand.new("move `#{source}' to `#{destination}'") do
command = "move `#{source}' to `#{destination}'"
build_commands << BuildCommand.new(command) do
Dir.chdir(software.project_dir) do
FileSyncer.glob(source).each do |file|
FileUtils.mv(file, destination, options)
files = FileSyncer.glob(source)
if files.empty?
log.warn(log_key) {"no matched files for glob #{command}"}
else
files.each do |file|
FileUtils.mv(file, destination, options)
end
end
end
end
......@@ -534,10 +531,16 @@ module Omnibus
# @return (see #command)
#
def link(source, destination, options = {})
build_commands << BuildCommand.new("link `#{source}' to `#{destination}'") do
command = "link `#{source}' to `#{destination}'"
build_commands << BuildCommand.new(command) do
Dir.chdir(software.project_dir) do
FileSyncer.glob(source).each do |file|
FileUtils.ln_s(file, destination, options)
files = FileSyncer.glob(source)
if files.empty?
log.warn(log_key) {"no matched files for glob #{command}"}
else
files.each do |file|
FileUtils.ln_s(file, destination, options)
end
end
end
end
......@@ -581,7 +584,7 @@ module Omnibus
def build
log.info(log_key) { 'Starting build' }
shasum # ensure shashum is calculated before build since the build can alter the shasum
log.internal(log_key) { 'Cached builder checksum before build: #{shasum}'}
log.internal(log_key) { "Cached builder checksum before build: #{shasum}"}
if software.overridden?
log.info(log_key) do
"Version overridden from #{software.default_version} to "\
......
......@@ -85,7 +85,7 @@ module Omnibus
if @options[:output_manifest]
FileUtils.mkdir_p('pkg')
File.open(::File.join('pkg', 'version-manifest.json'), 'w') do |f|
f.write(JSON.pretty_generate(project.built_manifest.to_hash))
f.write(project.built_manifest.to_json)
end
end
end
......
......@@ -66,37 +66,84 @@ module Omnibus
desc: "Explicit version for this changelog",
type: :string
desc 'generate', 'Generate a changelog for a new release'
def generate
g = GitRepository.new
if @options[:skip_components]
diff = Omnibus::EmptyManifestDiff.new
else
old_manifest = if @options[:starting_manifest]
Omnibus::Manifest.from_file(@options[:starting_manifest])
else
Omnibus::Manifest.from_hash(JSON.parse(g.file_at_revision("version-manifest.json",
g.latest_tag)))
end
new_manifest = Omnibus::Manifest.from_file(@options[:ending_manifest])
diff = Omnibus::ManifestDiff.new(old_manifest, new_manifest)
end
new_version = if @options[:version]
@options[:version]
elsif @options[:patch]
Omnibus::SemanticVersion.new(g.latest_tag).next_patch.to_s
elsif @options[:minor] && !@options[:major] # minor is the default so it will always be true
Omnibus::SemanticVersion.new(g.latest_tag).next_minor.to_s
elsif @options[:major]
Omnibus::SemanticVersion.new(g.latest_tag).next_major.to_s
end
desc 'generate [START] [END]', 'Generate a changelog for a new release'
def generate(start_ref=nil, end_ref=nil)
@start_ref = start_ref
@end_ref = end_ref
diff = if @options[:skip_components]
Omnibus::EmptyManifestDiff.new
else
Omnibus::ManifestDiff.new(old_manifest, new_manifest)
end
Omnibus::ChangeLogPrinter.new(ChangeLog.new(),
Omnibus::ChangeLogPrinter.new(ChangeLog.new(starting_revision, ending_revision),
diff,
@options[:source_path]).print(new_version)
end
private
def local_git_repo
GitRepository.new
end
def old_manifest
@old_manifest ||= if @options[:starting_manifest]
Omnibus::Manifest.from_file(@options[:starting_manifest])
else
manifest_for_revision(starting_revision)
end
end
def new_manifest
@new_manifest ||= if @options[:ending_manifest]
Omnibus::Manifest.from_file(@options[:ending_manifest])
else
manifest_for_revision(ending_revision)
end
end
def manifest_for_revision(rev)
Omnibus::Manifest.from_hash(JSON.parse(local_git_repo.file_at_revision("version-manifest.json", rev)))
end
def new_version
if @options[:version]
@options[:version]
elsif @options[:patch]
Omnibus::SemanticVersion.new(local_git_repo.latest_tag).next_patch.to_s
elsif @options[:minor] && !@options[:major] # minor is the default so it will always be true
Omnibus::SemanticVersion.new(local_git_repo.latest_tag).next_minor.to_s
elsif @options[:major]
Omnibus::SemanticVersion.new(local_git_repo.latest_tag).next_major.to_s
elsif @options[:ending_manifest]
new_manifest.build_version
end
end
# starting_revision is taken from:
# - value passed as the first argument
# - value found in the starting manifest
# - the latest git tag in the local repository
def starting_revision
@start_ref ||= if @options[:starting_manifest]
old_manifest.build_git_revision
else
local_git_repo.latest_tag
end
end
# ending_revision is taken from:
# - value passed as the first argument
# - value found in the ending manifest
# - HEAD in the current git repository
def ending_revision
@end_ref ||= if @options[:ending_manifest]
new_manifest.build_git_revision
else
'HEAD'
end
end
end
end
......@@ -18,19 +18,27 @@ module Omnibus
class Command::Publish < Command::Base
namespace :publish
# These options are useful for publish packages that were built for a
# paticluar platform/version and tested on another platform/version.
# This option is useful for publish packages that were built for a
# particular platform/version but tested on other platform/versions.
#
# For example, one might build on Ubuntu 10.04 and test/publish on
# Ubuntu 11.04, 12.04 and Debian 7.
# Ubuntu 10.04, 12.04, and 14.04.
#
# If these options are used with the glob pattern support all packages
# will be published to the same platform/version.
class_option :platform,
desc: 'The platform to publish for',
# @example JSON
# {
# "ubuntu-10.04": [
# "ubuntu-10.04",
# "ubuntu-12.04",
# "ubuntu-14.04"
# ]
# }
#
class_option :platform_mappings,
desc: 'The optional platform mappings JSON file to publish with',
type: :string
class_option :platform_version,
desc: 'The platform version to publish for',
class_option :version_manifest,
desc: 'Path to the version-manifest.json file to publish with',
type: :string
#
......@@ -43,6 +51,10 @@ module Omnibus
desc: 'The accessibility of the uploaded packages',
enum: %w(public private),
default: 'private'
method_option :region,
type: :string,
desc: 'The region in which the bucket is located',
default: 'us-east-1'
desc 's3 BUCKET PATTERN', 'Publish to an S3 bucket'
def s3(bucket, pattern)
options[:bucket] = bucket
......@@ -54,6 +66,14 @@ module Omnibus
#
# $ omnibus publish artifactory libs-omnibus-local pkg/chef*
#
method_option :build_record,
type: :boolean,
desc: 'Optionally create an Artifactory build record for the published artifacts',
default: true
method_option :properties,
type: :hash,
desc: 'Properites to attach to published artifacts',
default: {}
desc 'artifactory REPOSITORY PATTERN', 'Publish to an Artifactory instance'
def artifactory(repository, pattern)
options[:repository] = repository
......@@ -68,8 +88,12 @@ module Omnibus
# @return [void]
#
def publish(klass, pattern, options)
if options[:platform_mappings]
options[:platform_mappings] = JSON.parse(File.read(File.expand_path(options[:platform_mappings])))
end
klass.publish(pattern, options) do |package|
say("Uploaded '#{package.name}'", :green)
say("Published '#{package.name}' for #{package.metadata[:platform]}-#{package.metadata[:platform_version]}", :green)
end
end
end
......
......@@ -289,6 +289,14 @@ module Omnibus
raise MissingRequiredAttribute.new(self, :s3_secret_key, "'EFGH5678'")
end
# The region of the S3 bucket you want to cache software artifacts in.
# Defaults to 'us-east-1'
#
# @return [String]
default(:s3_region) do
'us-east-1'
end
# --------------------------------------------------
# @!endgroup
#
......@@ -443,6 +451,26 @@ module Omnibus
# @return [String, nil]
default(:solaris_linker_mapfile, "files/mapfiles/solaris")
# Architecture to target when building on windows. This option
# should affect the bit-ness of Ruby and DevKit used, the platform of
# any MSIs generated and package dlls being downloaded.
#
# See the windows_arch_i386? software definition dsl
# methods.
#
# @return [:x86, :x64]
default(:windows_arch) do
if Ohai['kernel']['machine'] == 'x86_64'
Omnibus.logger.deprecated('Config') do
"windows_arch is defaulting to :x86. In Omnibus 5, it will " \
"default to :x64 if the machine architecture is x86_64. " \
"If you would like to continue building 32 bit packages, please "\
"manually set windows_arch in your omnibus.rb file to :x86."
end
end
:x86
end
# --------------------------------------------------
# @!endgroup
#
......
......@@ -87,7 +87,7 @@ EOH
def to_s
<<-EOH
Attempting to evaluate the template `#{@name}', but it was not found at any of
Attempting to evaluate the template `#{@template}', but it was not found at any of
the following locations:
#{@search_paths.map { |path| " #{path}" }.join("\n")}
......@@ -214,6 +214,18 @@ EOH
end
end
class ChecksumMissing < Error
def initialize(software)
super <<-EOH