Reset Pivotal User in CHEF

These are low level commands in chef, please backup and make sure you can restore before running these commands

if you see the following error message in chef like I did:

chef-server-ctl user-list
 ERROR: Failed to authenticate to as pivotal with key /tmp/latovip20191029-14643-1na6z6n
 Response:  Invalid signature for user or client 'pivotal'

These commands may help you solve the problem:

  • Create public key for pivotal user
openssl rsa -in /etc/opscode/pivotal.pem -pubout > /var/opt/opscode/postgresql/9.2/data/
  • Get pivotal authz id
echo "SELECT authz_id FROM auth_actor WHERE id = 1" | su -l opscode-pgsql -c 'psql bifrost -tA' | tr -d '\n' > /var/opt/opscode/postgresql/9.2/data/pivotal.authz_id
  • Delete pivotal user from postgresdb
echo "DELETE FROM users WHERE authz_id = pg_read_file('pivotal.authz_id');" | su -l opscode-pgsql -c 'psql opscode_chef'
  • Insert new pivotal user to postgresdb
echo "INSERT INTO users (id, authz_id, username, email, pubkey_version, public_key, serialized_object, last_updated_by, created_at, updated_at) VALUES (md5(random()::text), pg_read_file('pivotal.authz_id'), 'pivotal', '[email protected]', 0, pg_read_file(''), '{\"first_name\":\"Clark\",\"last_name\":\"Kent\",\"display_name\":\"Clark Kent\"}', pg_read_file('pivotal.authz_id'), LOCALTIMESTAMP, LOCALTIMESTAMP);" | su -l opscode-pgsql -c 'psql opscode_chef'

I took the commands from the following issue:

Run Chef Server Behind Nginx

  • vi /etc/opscode/chef-server.rb
server_name = "" 
api_fqdn server_name
bookshelf['vip'] = server_name
nginx['url'] = "https://#{server_name}"
nginx['server_name'] = server_name
nginx['client_max_body_size'] = "1000m"
  • run chef server reconfigure
chef-server-ctl reconfigure
  • Configure nginx like any virtual server with regular http traffic
  • done

How to install chef-server inside docker container

  • Create volumes for chef logs and data
docker volume create chef-log
docker volume create chef-data
  • Run centos/systemd container with privileged permissions and ipv6 disabled
docker run --sysctl net.ipv6.conf.all.disable_ipv6=1 --privileged --name chef-server-core -d -v chef-data:/var/opt/opscode -v chef-log:/var/log/opscode -p 80:80 -p 443:443 centos/systemd
  • Connect to your new container
docker exec -it chef-server-core bash
  • Download and install chef
curl -o chef-server-core.rpm
rpm -Uvh chef-server-core.rpm
yum install rsync crontabs which net-tools less -y
systemctl enable crond
systemctl start crond
chef-server-ctl reconfigure
  • Note: if you want to restore from a backup run the following command:
chef-server-ctl restore -t 60000 /tmp/chef-backup-date.tgz
  • Note: When I install it I got the following error after running chef-server-ctl reconfigure:
Recipe Compile Error in /var/opt/opscode/local-mode-cache/cookbooks/private-chef/attributes/default.rb

undefined method `[]' for nil:NilClass

Cookbook Trace:
  /var/opt/opscode/local-mode-cache/cookbooks/private-chef/attributes/default.rb:616:in `from_file'

Relevant File Content:

609:  default['private_chef']['postgresql']['db_superuser'] = 'opscode-pgsql'
610:  default['private_chef']['postgresql']['shell'] = "/bin/sh"
611:  default['private_chef']['postgresql']['home'] = "/var/opt/opscode/postgresql"
612:  default['private_chef']['postgresql']['user_path'] = "/opt/opscode/embedded/bin:/opt/opscode/bin:$PATH"
613:  default['private_chef']['postgresql']['vip'] = ""
614:  default['private_chef']['postgresql']['port'] = 5432
615:  # We want to listen on all the loopback addresses, because we can't control which one localhost resolves to.
616>> default['private_chef']['postgresql']['listen_address'] = node['network']['interfaces']['lo']['addresses'].keys.join(',')
617:  default['private_chef']['postgresql']['max_connections'] = 350
618:  default['private_chef']['postgresql']['keepalives_idle'] = 60
619:  default['private_chef']['postgresql']['keepalives_interval'] = 15
620:  default['private_chef']['postgresql']['keepalives_count'] = 2
621:  default['private_chef']['postgresql']['md5_auth_cidr_addresses'] = [ '', '::1/128' ]
622:  default['private_chef']['postgresql']['wal_level'] = "minimal"
623:  default['private_chef']['postgresql']['archive_mode'] = "off" # "cannot be enabled when wal_level is set to minimal"
624:  default['private_chef']['postgresql']['archive_command'] = ""
625:  default['private_chef']['postgresql']['archive_timeout'] = 0 # 0 is disabled.

System Info:
ruby=ruby 2.5.5p157 (2019-03-15 revision 67260) [x86_64-linux]

Running handlers:
Running handlers complete
Chef Infra Client failed. 0 resources updated in 03 seconds
[2019-08-08T05:39:53+00:00] FATAL: Stacktrace dumped to /var/opt/opscode/local-mode-cache/chef-stacktrace.out
[2019-08-08T05:39:53+00:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report
[2019-08-08T05:39:53+00:00] FATAL: NoMethodError: undefined method `[]' for nil:NilClass
  • If you get the same error you need to configure default[‘private_chef’][‘postgresql’][‘listen_address’]
vi /opt/opscode/embedded/cookbooks/private-chef/attributes/default.rb
default['private_chef']['postgresql']['listen_address'] = "localhost"

chef-server-ctl reconfigure
  • Note: In chef-server-core 12.15.7 I got the following error:
Your system has IPv6 enabled but its loopback interface has no IPv6

You must either pass `ipv6.disable=1` to your kernel command line,
to completely disable IPv6, or ensure the loopback interface has an
`::1` address by running

    sysctl net.ipv6.conf.lo.disable_ipv6=0
  • To fix this you need to run the following
echo 0 > /proc/sys/net/ipv6/conf/all/disable_ipv6
chef-server-ctl reconfigure
  • Install chef-manage
chef-server-ctl install chef-manage
chef-server-ctl reconfigure
chef-manage-ctl reconfigure --accept-license
  • Creat chef admin
chef-server-ctl user-create USER_NAME FIRST_NAME LAST_NAME EMAIL 'PASSWORD' --filename FILE_NAME
  • Create organization
chef-server-ctl org-create short_name 'full_organization_name' --association_user user_name --filename ORGANIZATION-validator.pem
  • you can browse to your server ip address to see chef-manage. https://server_ip

Create data bag for certificate

It is not secure and you should use something like vault but if you still want to migrate certificate file to data bag you can use the following:

  1. Replace new lines with /n
    cat fullchain.pem | tr '\n' '#' | sed 's/#/\\n/g'
  2. Do the same thing for your key file
    cat key.pem | tr '\n' '#' | sed 's/#/\\n/g'
  3. Insert the data to the data bag

Dynamically List Git Branches in Jenkins Parameter

Jenkins git parameter plugin doesn’t list git branches if you work with bitbucket or at least it didn’t work for me, so I had to find other solution.

To get my git branches dynamically in a parameter I’m using Active Choices Plug-in with two scripts: one groovy that return the results to jenkins and the other one is a wrapper in bash which the groovy script use to get a list of git branches (because I don’t really know groovy:))


Before using this workaround you need to configure git on your jenkins server

How To

  • create bash script that will list your git branches
vi /usr/local/bin/
git ls-remote --heads --tags ${GIT_URL} | awk -F" " '{print $NF}'
  • make sure the script is executable
chmod +x /usr/local/bin/
  • In jenkins job configuration add “Active Choices Reactive Parameter”
  • In the name field enter BRANCH (or what ever you want)
  • Click on Groovy script and enter the following script
tags = []
text = " https://user:[email protected]/project/repo_name.git".execute().text
text.eachLine { tags.push(it) }
return tags



  • In the “Source  Code Management” section in “Branch to build” enter ${BRANCH}


if you have better suggestions or better groovy script please write a comment


Continuous deployment with jenkins and chef

Jenkins is a great CI/CD open source tool that we are using to deploy our applications, and chef is configuration management tool that helps you build your infrastructure automatically. I use both of them to create CI/CD environment in our projects.

Why use CI/CD:

  1. check that your code compiling
  2. can detect errors and fix them earlier
  3. commit smaller units which help you to revert back more easily
  4. automated tests helps you find broken features
  5. can release in any time
  6. avoid human errors
  7. deploy with zero downtime
  8. revert back to previous version with one click
  9. and more…

Here I am going to use chef recipes to deploy jenkins artifacts, but you can use any configuration management or scripts to do the same job.

Continuous deployment configuration

For continuous deployment I installed jenkins pipeline plugin and build the following pipeline:


  1. First job (build-manual) triggered manually by clicking the run icon and it creates new pipeline in our build pipeline
  2. Second job (build-deb) create deb package automatically after a successful or stable build  of  build-manual
  3. Third job (deploy-all) is a script that do the following:
    1. get all app servers from chef
    2. for each server it run chef recipe that download the created deb package and install it
    3. restart app service
    4. check that the server is ready by running a check command and only when it ready the script continue to the next server

The deployment script let us upgrade version with 0 downtime, because we upgrade one server at a time so we always have running servers behind our load balancer.

what we earn from this deployment:

  1. zero downtime
  2. automatic deployment (avoid human errors)
  3. ability to revert back – we can trigger the deploy script at any time from an older pipeline and it will download the deb package from that pipeline and install it on the servers
  4. deployment board – we know who run each build and when we deploy each version

we still can upgrade the pipeline by adding integration tests job, code coverage job, deploy to staging jobs and more.

Maybe I will get to it someday 🙂

001. Install Mcollective


Mcollective is a framwork that help you run parallel jobs.

In this guide I will show how to install Mcollective server and client and use it to run puppet agent on remote servers

Tested On

OS: Amazon Linux AMI release 2012.03
Mcollective version: 2.2.1-1
Hardware: Virtual Machine (AWS AMI)

Install Mcollective

  • Install Mcollective server, client and activemq on the mcollective server
yum install mcollective mcollective-client activemq -y
  • Start AtiveMQ
service activemq start
  • Starting Mcollective
service mcollective start
  • Testing the server by running the ping command on it
mco ping

Install Mcollective agent

  • Install the Mcollective agent on the client
yum install mcollective
  • Confgiure Mcollective agent
vi /etc/mcollective/server.cfg
... = puppet
  • Start Mcollective
service mcollective start

Install puppet addon for Mcollective

  • Download and Install puppet addon
cd /usr/libexec/mcollective/mcollective
wget -O master.tar.gz
tar zxf master.tar.gz
cd mcollective-plugins-master/agent/
rsync -rvP puppetd/* /usr/libexec/mcollective/mcollective/
  • Restart Mcollective service
service mcollective restart

For more inforamation about Mcollective please visit