Commit dc6b89b2 authored by Marin Jankovski's avatar Marin Jankovski

Merge tag 'v4.1.0' of https://github.com/chef/omnibus into upgrade_omnibus_version

Version 4.1.0
parents 1b336f74 27d41edb
......@@ -83,6 +83,14 @@ Gemfile.lock
*.tmproject
tmtags
# avoid generic ctags files too
tags
gems.tags
gemtags
.tags
.gem.tags
.gemtags
.tags_sorted_by_file
### SublimeText ###
# workspace files are user-specific
......
rvm:
- 1.9.3
- 2.0.0
- 2.1
bundler_args: --jobs 7 --without docs local
bundler_args: "--jobs 7 --without docs local"
branches:
only:
- 2.0-stable
- 3.0-stable
- master
script: bundle exec rake travis:ci
notifications:
hipchat:
slack:
on_change: true
on_failure: true
on_success: false
on_pull_requests: false
rooms:
# Build Statuses
- secure: JdLOITSPHW5xk8eoFOmH1Js5PT6iFgswUG8fIqNq69ie2Qws2K58hPDR6HW2NBEsBb7dW1S5jT6V9RHhm0ykekvEJVa5AjebO3EsZa+Cu/VahFMg4DL+SGYgKlKrZQosd+EgVQQ3C9gj0dApGtKqf2Ej7RNESwCPY8SOduTC6d8=
# Release Engineering
- secure: PmOaqp2DFyy79VpCuvSYILblK6tP0eDa9fDG9X2j+wggoosBkmlGfnSxp3A2TXlUyK62mr6/B89dRkL2aDWTJ/gqaZ34Elx2rV5S2A5YJIx9stA2+4iXeSlOF2YMNKGwBVmF/kp/nNoG2FqUkUBrRurWAOXZnwxKIhQ7+kLVb/0=
secure: udLCoBl71xJBsW3P+YqZbA5xfaB8l4IIF6SKEXpjIUYNYyP6m77uoY9PZ/M/5EN1ojPXDxsZ6SiPEJifVpkX5Hqd0eu1MwWyRXvOwiDf/nkg2Y3mqwUQRb3LDYWZxPn+xp5mHePe3mvTICR1zH49yx7fdZs8Qvzhh3Lno/w/qZ4=
Omnibus CHANGELOG
=================
v4.1.0 (Unreleased)
v4.1.0 (September 1, 2015)
-------------------------
### New Features
- Allow semver prefixes that start with "v"
- Reset target_shasum in PathFetcher after every fetch, preventing erroneous GitCache misses for builds that produced local artifacts during the previous omnibus run.
- Allow users to specify options to the underlying FileSyncer in a Software definition
- Add CPPFLAGS in with_standard_compiler_flags
- Add a Fedora dist tag to package name and spec release
- Copy distribution.xml.erb also when creating project with --pkg-assets
- Manage version manifests with Omnibus. Omnibus now creates a version manifest in text and JSON.
- Retry failed downloads
- Support inclusion of email address in Wix template
- Allow solaris to use mapfiles
- A new `omnibus generate changelog` command generates an opinionated CHANGELOG entry based on metadata in commit messages.
- Add warnings on empty globs.
- Raspberry Pi 2 support
- Clone git repositories with --recursive flag
- Add ability to sign MSIs
- Support running Omnibus with 64-bit ruby on Windows
- Create an Artifactory build for published packages
- Add a "windows_arch" omnibus option to choose 32/64 bit builds.
- Set directory ownership/permissions to match the filesystem package.
- Allow Windows 10 builders
- Allow omnibus to build "bundled" Windows installers
- Set perms on control files per Ubuntu Software Center's lintian checks.
- Replace uber-s3 dependency with aws-sdk
### Bug Fixes
- Config.append_timestamp is now properly handled by the build_version
DSL. For some users, this may introduce a change in behavior. To
revert to the old behavior set append_timestamp to false in
`omnibus.rb` or use --override append_timestamp:false at the command
line.
- Do not memoize current_revision to avoid returning incorrect data. Memoizing current_revision was previously causing version_for_cache to return the pre-fetch revision of the software, leading us to erroneously restore an older revision from the cache.
- Clean up "files listed twice" warnings. This should allow signing RPMs on EL 7
- dpkg uses arch name ppc64el (instead of ppc64le) for little endian
- Override equality operator for ManifestEntry. Prior to this, we simply tested basic object identity, which would result in many "updated" dependencies
- Correctly determine git repository when running in a subdirectory
- Forcibly remove a non-empty project dir before cloning because if multiple projects are using the same maching for building, they might have two different software definitions that share the same name but a different source.
- Accomidate hardlinks properly
- When building spec file, handle path names that contain spaces
- Add libkvm and libprocstat to freebsd whitelist
- Properly render ERB templates that do not have variables
- Change git fetcher to correctly fetch when repo name is the same but branch/version is different.
### Potentially Breaking Changes
- `Omnibus::SemanticVersion.new` now raises `Omnibus::InvalidVersion` instead of `Omnibus::SemanticVersion::InvalidVersion`
- Cache Builder#shasum before Builder#build to ensure consistent result.
- Update chef-sugar to 3.0 which adds ppc64le support.
- The MSI packager now adds the architecture to the msi name. The file names go from `package_name-build_version-build_iteration` to `package_name-build_version-build_iteration-arch`.
v4.0.0 (December 15, 2014)
--------------------------
......
# Maintainers
This file lists how the Omnibus project is maintained. When making changes
to the system, this file tells you who needs to review your patch - you need a
simple majority of maintainers for the relevant subsystems to provide a :+1: on
your pull request. Additionally, you need to not receive a veto from a
Lieutenant or the Project Lead.
Check out
[How Chef is Maintained](https://github.com/chef/chef-rfc/blob/master/rfc030-maintenance-policy.md#how-the-project-is-maintained)
for details on the process, how to become a maintainer, lieutenant, or the
project lead.
## Project Lead
* [Seth Chisamore](https://github.com/schisamo)
## Maintainers
* [Daniel DeLeo](https://github.com/danielsdeleo)
* [Jay Mundrawala](https://github.com/jdmundrawala)
* [Lamont Granquist](https://github.com/lamont-granquist)
* [Scott Hain](https://github.com/scotthain)
* [Seth Vargo](http://github.com/sethvargo)
* [Steven Danna](https://github.com/stevendanna)
* [Yvonne Lam](http://github.com/yzl)
......@@ -24,7 +24,7 @@ Prerequisites
-------------
Omnibus is designed to run with a minimal set of prerequisites. You will need the following:
- Ruby 1.9+
- Ruby 2.0.0+
- Bundler
......@@ -274,6 +274,24 @@ omnibus manifest PROJECT -l warn
This will output a JSON-formatted manifest containing the resolved
version of every software definition.
Whitelisting Libraries
----------------------
Sometimes a platform has libraries that need to be whitelisted so the healthcheck
can pass. The whitelist found in the [healthcheck](https://github.com/chef/omnibus/blob/master/lib/omnibus/health_check.rb)
code comprises the minimal required for successful builds on supported platforms.
To add your own whitelisted library, simply add the a regex to your software
definition in your omnibus project as follows:
```
whitelist_file /libpcrecpp\.so\..+/
```
It is typically a good idea to add a conditional to whitelist based on the specific
platform that requires it.
*Warning: You should only add libraries to the whitelist that are guaranteed to
be on the system you install to; if a library comes from a non-default package
you should instead build it into the package.*
Changelog
---------
......
version: "{build}"
os: Windows Server 2012
platform:
- x64
environment:
matrix:
- ruby_version: "200"
clone_folder: c:\projects\omnibus
clone_depth: 1
skip_tags: true
branches:
only:
- master
install:
- SET PATH=C:\Ruby%ruby_version%\bin;C:\Ruby21%ruby_version:~3%\Devkit\mingw\bin;%PATH%
- echo %PATH%
- ruby --version
- gem --version
- gem install bundler --quiet --no-ri --no-rdoc
- bundler --version
- cp C:\Ruby21%ruby_version:~3%\Devkit\mingw\bin\bsdtar.exe C:\Ruby21%ruby_version:~3%\Devkit\mingw\bin\tar.exe
- appveyor DownloadFile http://curl.haxx.se/ca/cacert.pem -FileName C:\cacert.pem
- set SSL_CERT_FILE=C:\cacert.pem
build_script:
- bundle install
test_script:
- bundle exec rake unit
- bundle exec rake functional
- bundle exec rake acceptance
......@@ -22,7 +22,7 @@ repository) is
customize the location of the cache in the `omnibus.rb` config file
using the key `git_cache_dir`. For example:
git_cache_dir "/opt/ominbus-caches"
git_cache_dir "/opt/omnibus-caches"
## How It Works ##
......
Feature: omnibus publish
Scenario: Overriding publishing platform
* I run `omnibus publish artifactory fake * --platform debian`
Scenario: Providing platform mappings file
* I have a platform mappings file named "platform_mappings.json"
* I run `omnibus publish artifactory fake * --platform-mappings platform_mappings.json`
* the output should contain:
"""
Publishing platform has been overriden to 'debian'
"""
Scenario: Overriding publishing platform version
* I run `omnibus publish artifactory fake * --platform-version 7`
* the output should contain:
"""
Publishing platform version has been overriden to '7'
Publishing will be performed using provided platform mappings.
"""
require 'aruba/api'
Given(/^I have an omnibus project named "(.+)"$/) do |name|
create_dir(name)
create_directory(name)
cd(name)
write_file("config/projects/#{name}.rb", <<-EOH.gsub(/^ {4}/, ''))
......@@ -27,3 +27,15 @@ Given(/^I have an omnibus project named "(.+)"$/) do |name|
package_tmp './local/omnibus/pkg-tmp'
EOH
end
Given(/^I have a platform mappings file named "(.+)"$/) do |name|
write_file(name, <<-EOH.gsub(/^ {4}/, ''))
{
"ubuntu-10.04": [
"ubuntu-10.04",
"ubuntu-12.04",
"ubuntu-14.04"
]
}
EOH
end
......@@ -4,10 +4,12 @@ require 'aruba/in_process'
require 'omnibus/cli'
Aruba.configure do |config|
config.command_launcher = :in_process
config.main_class = Omnibus::CLI::Runner
end
Before do
# Reset anything that might have been cached in the Omnibus project
Omnibus.reset!(true)
Aruba::InProcess.main_class = Omnibus::CLI::Runner
Aruba.process = Aruba::InProcess
end
......@@ -488,18 +488,24 @@ module Omnibus
# @return (see #command)
#
def copy(source, destination, options = {})
build_commands << BuildCommand.new("copy `#{source}' to `#{destination}'") do
command = "copy `#{source}' to `#{destination}'"
build_commands << BuildCommand.new(command) do
Dir.chdir(software.project_dir) do
FileSyncer.glob(source).each do |file|
files = FileSyncer.glob(source)
if files.empty?
log.warn(log_key) {"no matched files for glob #{command}"}
else
files.each do |file|
FileUtils.cp_r(file, destination, options)
end
end
end
end
end
expose :copy
#
# Copy the given source to the destination. This method accepts a single
# Move the given source to the destination. This method accepts a single
# file or a file pattern to match
#
# @param [String] source
......@@ -511,14 +517,20 @@ module Omnibus
# @return (see #command)
#
def move(source, destination, options = {})
build_commands << BuildCommand.new("move `#{source}' to `#{destination}'") do
command = "move `#{source}' to `#{destination}'"
build_commands << BuildCommand.new(command) do
Dir.chdir(software.project_dir) do
FileSyncer.glob(source).each do |file|
files = FileSyncer.glob(source)
if files.empty?
log.warn(log_key) {"no matched files for glob #{command}"}
else
files.each do |file|
FileUtils.mv(file, destination, options)
end
end
end
end
end
expose :move
#
......@@ -534,14 +546,20 @@ module Omnibus
# @return (see #command)
#
def link(source, destination, options = {})
build_commands << BuildCommand.new("link `#{source}' to `#{destination}'") do
command = "link `#{source}' to `#{destination}'"
build_commands << BuildCommand.new(command) do
Dir.chdir(software.project_dir) do
FileSyncer.glob(source).each do |file|
files = FileSyncer.glob(source)
if files.empty?
log.warn(log_key) {"no matched files for glob #{command}"}
else
files.each do |file|
FileUtils.ln_s(file, destination, options)
end
end
end
end
end
expose :link
#
......@@ -581,7 +599,7 @@ module Omnibus
def build
log.info(log_key) { 'Starting build' }
shasum # ensure shashum is calculated before build since the build can alter the shasum
log.internal(log_key) { 'Cached builder checksum before build: #{shasum}'}
log.internal(log_key) { "Cached builder checksum before build: #{shasum}"}
if software.overridden?
log.info(log_key) do
"Version overridden from #{software.default_version} to "\
......
......@@ -85,7 +85,7 @@ module Omnibus
if @options[:output_manifest]
FileUtils.mkdir_p('pkg')
File.open(::File.join('pkg', 'version-manifest.json'), 'w') do |f|
f.write(JSON.pretty_generate(project.built_manifest.to_hash))
f.write(project.built_manifest.to_json)
end
end
end
......
......@@ -66,37 +66,84 @@ module Omnibus
desc: "Explicit version for this changelog",
type: :string
desc 'generate', 'Generate a changelog for a new release'
def generate
g = GitRepository.new
if @options[:skip_components]
diff = Omnibus::EmptyManifestDiff.new
desc 'generate [START] [END]', 'Generate a changelog for a new release'
def generate(start_ref=nil, end_ref=nil)
@start_ref = start_ref
@end_ref = end_ref
diff = if @options[:skip_components]
Omnibus::EmptyManifestDiff.new
else
old_manifest = if @options[:starting_manifest]
Omnibus::ManifestDiff.new(old_manifest, new_manifest)
end
Omnibus::ChangeLogPrinter.new(ChangeLog.new(starting_revision, ending_revision),
diff,
@options[:source_path]).print(new_version)
end
private
def local_git_repo
GitRepository.new
end
def old_manifest
@old_manifest ||= if @options[:starting_manifest]
Omnibus::Manifest.from_file(@options[:starting_manifest])
else
Omnibus::Manifest.from_hash(JSON.parse(g.file_at_revision("version-manifest.json",
g.latest_tag)))
manifest_for_revision(starting_revision)
end
new_manifest = Omnibus::Manifest.from_file(@options[:ending_manifest])
diff = Omnibus::ManifestDiff.new(old_manifest, new_manifest)
end
new_version = if @options[:version]
def new_manifest
@new_manifest ||= if @options[:ending_manifest]
Omnibus::Manifest.from_file(@options[:ending_manifest])
else
manifest_for_revision(ending_revision)
end
end
def manifest_for_revision(rev)
Omnibus::Manifest.from_hash(JSON.parse(local_git_repo.file_at_revision("version-manifest.json", rev)))
end
def new_version
if @options[:version]
@options[:version]
elsif @options[:patch]
Omnibus::SemanticVersion.new(g.latest_tag).next_patch.to_s
Omnibus::SemanticVersion.new(local_git_repo.latest_tag).next_patch.to_s
elsif @options[:minor] && !@options[:major] # minor is the default so it will always be true
Omnibus::SemanticVersion.new(g.latest_tag).next_minor.to_s
Omnibus::SemanticVersion.new(local_git_repo.latest_tag).next_minor.to_s
elsif @options[:major]
Omnibus::SemanticVersion.new(g.latest_tag).next_major.to_s
Omnibus::SemanticVersion.new(local_git_repo.latest_tag).next_major.to_s
elsif @options[:ending_manifest]
new_manifest.build_version
end
end
# starting_revision is taken from:
# - value passed as the first argument
# - value found in the starting manifest
# - the latest git tag in the local repository
def starting_revision
@start_ref ||= if @options[:starting_manifest]
old_manifest.build_git_revision
else
local_git_repo.latest_tag
end
end
Omnibus::ChangeLogPrinter.new(ChangeLog.new(),
diff,
@options[:source_path]).print(new_version)
# ending_revision is taken from:
# - value passed as the first argument
# - value found in the ending manifest
# - HEAD in the current git repository
def ending_revision
@end_ref ||= if @options[:ending_manifest]
new_manifest.build_git_revision
else
'HEAD'
end
end
end
end
......@@ -18,19 +18,27 @@ module Omnibus
class Command::Publish < Command::Base
namespace :publish
# These options are useful for publish packages that were built for a
# paticluar platform/version and tested on another platform/version.
# This option is useful for publish packages that were built for a
# particular platform/version but tested on other platform/versions.
#
# For example, one might build on Ubuntu 10.04 and test/publish on
# Ubuntu 11.04, 12.04 and Debian 7.
# Ubuntu 10.04, 12.04, and 14.04.
#
# If these options are used with the glob pattern support all packages
# will be published to the same platform/version.
class_option :platform,
desc: 'The platform to publish for',
# @example JSON
# {
# "ubuntu-10.04": [
# "ubuntu-10.04",
# "ubuntu-12.04",
# "ubuntu-14.04"
# ]
# }
#
class_option :platform_mappings,
desc: 'The optional platform mappings JSON file to publish with',
type: :string
class_option :platform_version,
desc: 'The platform version to publish for',
class_option :version_manifest,
desc: 'Path to the version-manifest.json file to publish with',
type: :string
#
......@@ -43,6 +51,10 @@ module Omnibus
desc: 'The accessibility of the uploaded packages',
enum: %w(public private),
default: 'private'
method_option :region,
type: :string,
desc: 'The region in which the bucket is located',
default: 'us-east-1'
desc 's3 BUCKET PATTERN', 'Publish to an S3 bucket'
def s3(bucket, pattern)
options[:bucket] = bucket
......@@ -54,6 +66,10 @@ module Omnibus
#
# $ omnibus publish artifactory libs-omnibus-local pkg/chef*
#
method_option :build_record,
type: :boolean,
desc: 'Optionally create an Artifactory build record for the published artifacts',
default: true
desc 'artifactory REPOSITORY PATTERN', 'Publish to an Artifactory instance'
def artifactory(repository, pattern)
options[:repository] = repository
......@@ -68,8 +84,12 @@ module Omnibus
# @return [void]
#
def publish(klass, pattern, options)
if options[:platform_mappings]
options[:platform_mappings] = JSON.parse(File.read(File.expand_path(options[:platform_mappings])))
end
klass.publish(pattern, options) do |package|
say("Uploaded '#{package.name}'", :green)
say("Published '#{package.name}' for #{package.metadata[:platform]}-#{package.metadata[:platform_version]}", :green)
end
end
end
......
......@@ -289,6 +289,14 @@ module Omnibus
raise MissingRequiredAttribute.new(self, :s3_secret_key, "'EFGH5678'")
end
# The region of the S3 bucket you want to cache software artifacts in.
# Defaults to 'us-east-1'
#
# @return [String]
default(:s3_region) do
'us-east-1'
end
# --------------------------------------------------
# @!endgroup
#
......@@ -443,6 +451,26 @@ module Omnibus
# @return [String, nil]
default(:solaris_linker_mapfile, "files/mapfiles/solaris")
# Architecture to target when building on windows. This option
# should affect the bit-ness of Ruby and DevKit used, the platform of
# any MSIs generated and package dlls being downloaded.
#
# See the windows_arch_i386? software definition dsl
# methods.
#
# @return [:x86, :x64]
default(:windows_arch) do
if Ohai['kernel']['machine'] == 'x86_64'
Omnibus.logger.deprecated('Config') do
"windows_arch is defaulting to :x86. In Omnibus 5, it will " \
"default to :x64 if the machine architecture is x86_64. " \
"If you would like to continue building 32 bit packages, please "\
"manually set windows_arch in your omnibus.rb file to :x86."
end
end
:x86
end
# --------------------------------------------------
# @!endgroup
#
......
......@@ -87,7 +87,7 @@ EOH
def to_s
<<-EOH
Attempting to evaluate the template `#{@name}', but it was not found at any of
Attempting to evaluate the template `#{@template}', but it was not found at any of
the following locations:
#{@search_paths.map { |path| " #{path}" }.join("\n")}
......@@ -317,4 +317,10 @@ EOH
EOF
end
end
class FailedToTimestampMSI < Error
def initialize
super("Failed to add timestamp to MSI.")
end
end
end
......@@ -36,13 +36,24 @@ module Omnibus
attr_reader :source
#
# The exact upstream version that a fetcher should fetch. For
# sources that allow aliases (branch name, tags, etc). Users
# The exact upstream version that a fetcher should fetch.
#
# @return [String]
#
# For sources that allow aliases (branch name, tags, etc). Users
# should use the class method resolve_version to determine this
# before constructing a fetcher.
#
attr_reader :resolved_version
#
# The upstream version as described before resolution.
#
# @return [String]
#
# This will usually be the same as +resolved_version+ but may
# refer toa remote ref name or tag for a source such as git.
attr_reader :described_version
#
# The path where extracted software should live.
#
......@@ -64,6 +75,7 @@ module Omnibus
@name = manifest_entry.name
@source = manifest_entry.locked_source
@resolved_version = manifest_entry.locked_version
@described_version = manifest_entry.described_version
@project_dir = project_dir
@build_dir = build_dir
end
......
......@@ -62,10 +62,11 @@ module Omnibus
create_required_directories
if cloned?
git_fetch(resolved_version) unless same_revision?(resolved_version)
git_fetch unless same_revision?(resolved_version)
else
force_recreate_project_dir! unless dir_empty?(project_dir)
git_clone
git_checkout(resolved_version)
git_checkout
end
end
......@@ -90,6 +91,33 @@ module Omnibus
source[:git]
end
#
# Determine if submodules should be cloned.
#
# @return [true, false]
#
def clone_submodules?
source[:submodules] || false
end
#
# Determine if a directory is empty
#
# @return [true, false]
#
def dir_empty?(dir)
Dir.entries(dir).reject {|d| ['.', '..'].include?(d) }.empty?
end