Commit e33729be authored by Serdar Sutay's avatar Serdar Sutay Committed by GitHub

Merge pull request #705 from chef/sersut/transitive-dep-pr

Include licensing information for transitive dependencies
parents d2a7b711 d747862b
Omnibus CHANGELOG
=================
v5.5.0 (August 16, 2016)
-----------------------
### New Features
- Add build timings to the local output (#669)
- Add `appx` packager (#675, #676, #677)
- `fatal_licensing_warnings` configuration to fail the build when there are licensing problems (#678)
- Ensure consistent code style using Chefstyle \o/ (#681)
- Drop Ruby 2.0 support (#697)
- Remove fakeroot from RPM packager (#698)
- Support license collection with git cache (#700)
- Add `SERIAL_NUMBER` to omnibus code, which is used in git caching to invalidate caches when omnibus code changes require it to be invalidated (#704)
- Support license collection of transitive dependencies (via license_scout gem) (#705)
- Add `skip_transitive_dependency_licensing` dsl method to Software. Use this when the software does not use any dependency manager. (#705)
### Bug Fixes
- Make license files readable inside the packages (#673)
- Prefer server marketing names on windows (#679)
- Instead of exe, use bat if present on windows (#684)
- Fix changelog generation with symbolized keys (#687)
v5.4.0 (April 18, 2016)
-----------------------
### New Features
......
......@@ -4,6 +4,9 @@ gemspec
# Fork to allow for a recent version of multipart-post.
gem "pedump", git: "https://github.com/ksubrama/pedump", branch: "patch-1"
# Always use license_scout from master
gem "license_scout", github: "chef/license_scout"
group :docs do
gem "yard", "~> 0.8"
gem "redcarpet", "~> 2.2.2"
......
......@@ -753,10 +753,6 @@ module Omnibus
private
def embedded_bin(bin)
windows_safe_path("#{install_dir}/embedded/bin/#{bin}")
end
#
# The **in-order** list of {BuildCommand} for this builder.
#
......
......@@ -514,6 +514,12 @@ module Omnibus
# @return [true, false]
default(:fatal_licensing_warnings, false)
# Fail the build or warn when build encounters a transitive dependency
# licensing warning.
#
# @return [true, false]
default(:fatal_transitive_dependency_licensing_warnings, false)
# --------------------------------------------------
# @!endgroup
#
......
......@@ -22,6 +22,18 @@ module Omnibus
include Util
include Logging
# The serial number represents compatibility of a cache entry with the
# current version of the omnibus code base. Any time a change is made to
# omnibus that makes the code incompatible with any cache entries created
# before the code change, the serial number should be incremented.
#
# For example, if a code change generates content in the `install_dir`
# before cache snapshots are taken, any snapshots created before upgrade
# will not have the generated content, so these snapshots would be
# incompatible with the current omnibus codebase. Incrementing the serial
# number ensures these old shapshots will not be used in subsequent builds.
SERIAL_NUMBER = 1
REQUIRED_GIT_FILES = %w{
HEAD
description
......@@ -98,7 +110,7 @@ refs}.freeze
# dependencies, including the on currently being acted upon.
shasums = [dep_list.map(&:shasum), software.shasum].flatten
suffix = Digest::SHA256.hexdigest(shasums.join("|"))
@tag = "#{software.name}-#{suffix}"
@tag = "#{software.name}-#{suffix}-#{SERIAL_NUMBER}"
log.internal(log_key) { "tag: #{@tag}" }
......
......@@ -17,6 +17,8 @@
require "uri"
require "fileutils"
require "omnibus/download_helpers"
require "license_scout/collector"
require "license_scout/options"
module Omnibus
class Licensing
......@@ -25,6 +27,7 @@ module Omnibus
include Sugarable
OUTPUT_DIRECTORY = "LICENSES".freeze
CACHE_DIRECTORY = "license-cache".freeze
class << self
......@@ -57,6 +60,7 @@ module Omnibus
yield license_collector
license_collector.process_transitive_dependency_licensing_info
license_collector.create_project_license_file
license_collector.raise_if_warnings_fatal!
end
......@@ -77,6 +81,21 @@ module Omnibus
#
attr_reader :licensing_warnings
#
# The warnings encountered while preparing the licensing information for
# transitive dependencies.
#
# @return [Array<String>]
#
attr_reader :transitive_dependency_licensing_warnings
#
# Manifest data of transitive dependency licensing information
#
# @return Hash
#
attr_reader :dep_license_map
#
# @param [Project] project
# the project to create licenses for.
......@@ -84,6 +103,8 @@ module Omnibus
def initialize(project)
@project = project
@licensing_warnings = []
@transitive_dependency_licensing_warnings = []
@dep_license_map = {}
end
#
......@@ -95,6 +116,9 @@ module Omnibus
FileUtils.rm_rf(output_dir)
FileUtils.mkdir_p(output_dir)
FileUtils.touch(output_dir_gitkeep_file)
FileUtils.rm_rf(cache_dir)
FileUtils.mkdir_p(cache_dir)
FileUtils.touch(cache_dir_gitkeep_file)
end
# Required callback to use instances of this class as a build wrapper for
......@@ -120,6 +144,10 @@ module Omnibus
#
def execute_post_build(software)
collect_licenses_for(software)
unless software.skip_transitive_dependency_licensing
collect_transitive_dependency_licenses_for(software)
end
end
#
......@@ -181,6 +209,8 @@ module Omnibus
f.puts project_license_content
f.puts ""
f.puts components_license_summary
f.puts ""
f.puts dependencies_license_summary
end
end
......@@ -227,6 +257,41 @@ module Omnibus
out
end
#
# Summary of the licenses of the transitive dependencies of the project.
# It is in the form of:
# ...
# This product includes inifile 3.0.0
# which is a 'ruby_bundler' dependency of 'chef',
# and which is available under a 'MIT' License.
# For details, see:
# /opt/opscode/LICENSES/ruby_bundler-inifile-3.0.0-README.md
# ...
#
# @return [String]
#
def dependencies_license_summary
out = "\n\n"
dep_license_map.each do |dep_mgr_name, data|
data.each do |dep_name, data|
data.each do |dep_version, dep_data|
projects = dep_data["dependency_of"].sort.map { |p| "'#{p}'" }.join(", ")
files = dep_data["license_files"].map { |f| File.join(output_dir, f) }
out << "This product includes #{dep_name} #{dep_version}\n"
out << "which is a '#{dep_mgr_name}' dependency of #{projects},\n"
out << "and which is available under a '#{dep_data["license"]}' License.\n"
out << "For details, see:\n"
out << files.join("\n")
out << "\n\n"
end
end
end
out
end
#
# Map that collects information about the licenses of the softwares
# included in the project.
......@@ -303,6 +368,24 @@ module Omnibus
File.join(output_dir, ".gitkeep")
end
# Cache directory where transitive dependency licenses will be collected in.
#
# @return [String]
#
def cache_dir
File.expand_path(CACHE_DIRECTORY, project.install_dir)
end
#
# Path to a .gitkeep file we create in the cache dir so git caching
# doesn't delete the directory.
#
# @return [String]
#
def cache_dir_gitkeep_file
File.join(cache_dir, ".gitkeep")
end
#
# Returns if the given path to a license is local or a remote url.
#
......@@ -337,14 +420,125 @@ module Omnibus
log.warn(log_key) { message }
end
#
# Logs the given message as warning or fails the build depending on the
# :fatal_transitive_dependency_licensing_warnings configuration setting.
#
# @param [String] message
# message to log as warning
def transitive_dependency_licensing_warning(message)
transitive_dependency_licensing_warnings << message
log.warn(log_key) { message }
end
def raise_if_warnings_fatal!
warnings_to_raise = []
if Config.fatal_licensing_warnings && !licensing_warnings.empty?
raise LicensingError.new(licensing_warnings)
warnings_to_raise << licensing_warnings
end
if Config.fatal_transitive_dependency_licensing_warnings && !transitive_dependency_licensing_warnings.empty?
warnings_to_raise << transitive_dependency_licensing_warnings
end
warnings_to_raise.flatten!
raise LicensingError.new(warnings_to_raise) unless warnings_to_raise.empty?
end
# 1. Parse all the licensing information for all software from 'cache_dir'
# 2. Merge and drop the duplicates
# 3. Add these licenses to the main manifest, to be merged with the main
# licensing information from software definitions.
def process_transitive_dependency_licensing_info
Dir.glob("#{cache_dir}/*/*-dependency-licenses.json").each do |license_manifest_path|
license_manifest_data = FFI_Yajl::Parser.parse(File.read(license_manifest_path))
project_name = license_manifest_data["project_name"]
dependency_license_dir = File.dirname(license_manifest_path)
license_manifest_data["dependency_managers"].each do |dep_mgr_name, dependencies|
dep_license_map[dep_mgr_name] ||= {}
dependencies.each do |dependency|
# Copy dependency files
dependency["license_files"].each do |f|
license_path = File.join(dependency_license_dir, f)
output_path = File.join(output_dir, f)
FileUtils.cp(license_path, output_path)
end
dep_name = dependency["name"]
dep_version = dependency["version"]
# If we already have this dependency we do not need to add it again.
if dep_license_map[dep_mgr_name][dep_name] && dep_license_map[dep_mgr_name][dep_name][dep_version]
dep_license_map[dep_mgr_name][dep_name][dep_version]["dependency_of"] << project_name
else
dep_license_map[dep_mgr_name][dep_name] ||= {}
dep_license_map[dep_mgr_name][dep_name][dep_version] = {
"license" => dependency["license"],
"license_files" => dependency["license_files"],
"dependency_of" => [ project_name ],
}
end
end
end
end
FileUtils.rm_rf(cache_dir)
end
private
# Uses license_scout to collect the licenses for transitive dependencies
# into #{output_dir}/license-cache/#{software.name}
def collect_transitive_dependency_licenses_for(software)
# We collect the licenses of the transitive dependencies of this software
# with LicenseScout. We place these files under
# /opt/project-name/license-cache for them to be cached in git_cache. Once
# the build completes we will process these license files but we need to
# perform this step after build, before git_cache to be able to operate
# correctly with the git_cache.
license_output_dir = File.join(cache_dir, software.name)
collector = LicenseScout::Collector.new(
software.project.name,
software.project_dir,
license_output_dir,
LicenseScout::Options.new(
environment: software.with_embedded_path,
ruby_bin: software.embedded_bin("ruby")
)
)
begin
collector.run
collector.issue_report.each { |i| transitive_dependency_licensing_warning(i) }
rescue LicenseScout::Exceptions::UnsupportedProjectType => e
# Looks like this project is not supported by LicenseScout. Either the
# language and the dependency manager used by the project is not
# supported, or the software definition does not have any transitive
# dependencies. In the latter case software definition should set
# 'skip_transitive_dependency_licensing' to 'true' to correct this
# error.
transitive_dependency_licensing_warning(<<-EOH)
Software '#{software.name}' is not supported project type for transitive \
dependency license collection. See https://github.com/chef/license_scout for \
the list of supported languages and dependency managers. If this project does \
not have any transitive dependencies, consider setting \
'skip_transitive_dependency_licensing' to 'true' in order to correct this error.
EOH
rescue LicenseScout::Exceptions::Error => e
transitive_dependency_licensing_warning(<<-EOH)
Can not automatically detect licensing information for '#{software.name}' using \
license_scout. Error is: '#{e}'
EOH
rescue Exception => e
transitive_dependency_licensing_warning(<<-EOH)
Unexpected error while running license_scout for '#{software.name}': '#{e}'
EOH
end
end
# Collect the license files for the software.
def collect_licenses_for(software)
return nil if software.license == :project_license
......
......@@ -76,6 +76,7 @@ module Omnibus
include Logging
include NullArgumentable
include Sugarable
include Util
attr_reader :manifest
......@@ -378,6 +379,26 @@ module Omnibus
end
expose :license_file
#
# Skip collecting licenses of transitive dependencies for this software
#
# @example
# skip_transitive_dependency_licensing true
#
# @param [Boolean] val
# set or reset transitive dependency license collection
#
# @return [Boolean]
#
def skip_transitive_dependency_licensing(val = NULL)
if null?(val)
@skip_transitive_dependency_licensing || false
else
@skip_transitive_dependency_licensing = val
end
end
expose :skip_transitive_dependency_licensing
#
# Evaluate a block only if the version matches.
#
......@@ -736,6 +757,20 @@ module Omnibus
end
expose :with_embedded_path
#
# Returns the platform safe full path under embedded/bin directory to the
# given binary
#
# @param [String] bin
# Name of the binary under embedded/bin
#
# @return [String]
#
def embedded_bin(bin)
windows_safe_path("#{install_dir}/embedded/bin/#{bin}")
end
expose :embedded_bin
#
# A PATH variable format string representing the current PATH with the
# given path prepended. The correct path separator
......
......@@ -15,5 +15,5 @@
#
module Omnibus
VERSION = "5.4.0"
VERSION = "5.5.0"
end
......@@ -33,6 +33,7 @@ Gem::Specification.new do |gem|
gem.add_dependency "aws-sdk", "~> 2"
gem.add_dependency "thor", "~> 0.18"
gem.add_dependency "ffi-yajl", "~> 2.2"
gem.add_dependency "license_scout"
gem.add_development_dependency "bundler"
gem.add_development_dependency "artifactory", "~> 2.0"
......
inifile [![Build Status](https://secure.travis-ci.org/TwP/inifile.png)](http://travis-ci.org/TwP/inifile)
=======
This is a native Ruby package for reading and writing INI files.
Description
-----------
Although made popular by Windows, INI files can be used on any system thanks
to their flexibility. They allow a program to store configuration data, which
can then be easily parsed and changed. Two notable systems that use the INI
format are Samba and Trac.
More information about INI files can be found on the [Wikipedia Page](http://en.wikipedia.org/wiki/INI_file).
### Properties
The basic element contained in an INI file is the property. Every property has
a name and a value, delimited by an equals sign *=*. The name appears to the
left of the equals sign and the value to the right.
name=value
### Sections
Section declarations start with *[* and end with *]* as in `[section1]` and
`[section2]` shown in the example below. The section declaration marks the
beginning of a section. All properties after the section declaration will be
associated with that section.
### Comments
All lines beginning with a semicolon *;* or a number sign *#* are considered
to be comments. Comment lines are ignored when parsing INI files.
### Example File Format
A typical INI file might look like this:
[section1]
; some comment on section1
var1 = foo
var2 = doodle
var3 = multiline values \
are also possible
[section2]
# another comment
var1 = baz
var2 = shoodle
Implementation
--------------
The format of INI files is not well defined. Several assumptions are made by
the **inifile** gem when parsing INI files. Most of these assumptions can be
modified at, but the defaults are listed below.
### Global Properties
If the INI file lacks any section declarations, or if there are properties
decalared before the first section, then these properties will be placed into
a default "global" section. The name of this section can be configured when
creating an `IniFile` instance.
### Duplicate Properties
Duplicate properties are allowed in a single section. The last property value
set is the one that will be stored in the `IniFile` instance.
[section1]
var1 = foo
var2 = bar
var1 = poodle
The resulting value of `var1` will be `poodle`.
### Duplicate Sections
If you have more than one section with the same name then the sections will be
merged. Duplicate properties between the two sections will follow the rules
discussed above. Properties in the latter section will override properties in
the earlier section.
### Comments
The comment character can be either a semicolon *;* or a number sign *#*. The
comment character can appear anywhere on a line including at the end of a
name/value pair declaration. If you wish to use a comment character in your
value then you will need to either escape the character or put the value in
double quotations.
[section1]
var1 = foo # a comment
var2 = "foo # this is not a comment"
var3 = foo \# this is not a comment either
### Multi-Line Values
Values can be continued onto multiple lines in two separate ways. Putting a
slash at the end of a line will continue the value declaration to the next
line. When parsing, the trailing slash will be consumed and **will not**
appear in the resulting value. Comments can appear to the right of the
trailing slash.
var1 = this is a \ # these comments will
multiline value # be ignored by the parser
In the above example the resulting value for `var1` will be `this is a
multiline value`. If you want to preserve newline characters in the value then
quotations should be used.
var2 = "this is a
multiline value"
The resulting value for `var2` will be `this is a\nmultiline value`.
### Escape Characters
Several escape characters are supported within the **value** for a property.
These escape sequences will be applied to quoted and unquoted values alike.
You can enable or disable escaping by setting the **escape** flag to true or
false when creating an IniFile instance.
* \0 -- null character
* \n -- newline character
* \r -- carriage return character
* \t -- tab character
* \\\\ -- backslash character
The backslash escape sequence is only needed if you want one of the escape
sequences to appear literally in your value. For example:
property = this is not a tab \\t character
### Value Type Casting
Some values will be type cast when parsed by the code. Those values are
booleans, integers, floats, and empty strings are cast to `nil`.
* "" --> nil
* "42" --> 42
* "3.14159" --> 3.14159
* "true" --> true
* "false" --> false
* "normal string" --> "normal string"
Pretty basic stuff.
Install
-------
gem install inifile
Testing
-------
To run the tests:
$ rake
Examples
--------
require 'inifile'
myini = IniFile.load('mytest.ini')
myini.each_section do |section|
puts "I want #{myini[section]['somevar']} printed here!"
end
Contributing
------------
Contributions are gladly welcome! For small modifications (fixing typos,
improving documentation) you can use GitHub's in-browser editing capabilities
to create a pull request. For larger modifications I would recommend forking
the project, creating your patch, and then submitting a pull request.
Mr Bones is used to manage rake tasks and to install dependent files. To setup
your environment ...
$ gem install bones
$ rake gem:install_dependencies
And always remember that `rake -T` will show you the list of available tasks.
License
-------
MIT License
Copyright (c) 2006 - 2014
Permission is hereby granted, free of charge, to any person obtaining
a copy of this software and associated documentation files (the
'Software'), to deal in the Software without restriction, including
without limitation the rights to use, copy, modify, merge, publish,
distribute, sublicense, and/or sell copies of the Software, and to
permit persons to whom the Software is furnished to do so, subject to
the following conditions:
The above copyright notice and this permission notice shall be
included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
{
"license_manifest_version": 1,
"project_name": "snoopy",
"dependency_managers": {
"ruby_bundler": [
{
"name": "inifile",
"version": "3.0.0",
"license": "MIT",
"license_files": [
"ruby_bundler-inifile-3.0.0-README.md"
]
},
{
"name": "bundler-audit",
"version": "0.5.0",
"license": "GPLv3",
"license_files": [
"ruby_bundler-bundler-audit-0.5.0-COPYING.txt"
]
}
]
}
}
inifile [![Build Status](https://secure.travis-ci.org/TwP/inifile.png)](http://travis-ci.org/TwP/inifile)
=======