I have a Vagrant CentOS VM running with ps.memory = 2048 RAM allocated.
When I try to start the puppetserver service:
$ puppet --version 4.4.0 $ sudo puppet resource service puppetserver ensure=running Error: Could not start Service[puppetserver]: Execution of '/bin/systemctl start puppetserver' returned 1: Job for puppetserver.service failed. See 'systemctl status puppetserver.service' and 'journalctl -xn' for details. Error: /Service[puppetserver]/ensure: change from stopped to running failed: Could not start Service[puppetserver]: Execution of '/bin/systemctl start puppetserver' returned 1: Job for puppetserver.service failed. See 'systemctl status puppetserver.service' and 'journalctl -xn' for details. service { 'puppetserver': ensure => 'stopped', } $ journalctl -xn No journal files were found. $ systemctl status puppetserver.service puppetserver.service - puppetserver Service Loaded: loaded (/usr/lib/systemd/system/puppetserver.service; disabled) Process: 4708 ExecStartPre=/usr/bin/install --directory --owner=puppet --group=puppet --mode=775 /var/run/puppetlabs/puppetserver (code=exited, status=0/SUCCESS) Main PID: 4709 (java); : 4710 (bash) CGroup: /system.slice/puppetserver.service ├─4709 /usr/bin/java -Xms1g -Xmx1g -XX:MaxPermSize=1g -XX:OnOutOfMemoryError=kill -9 %p -Djava.security.egd=/... └─control ├─4710 /bin/bash /opt/puppetlabs/server/apps/puppetserver/ezbake-functions.sh wait_for_app └─4755 sleep 1 My JAVA_ARGS from /etc/sysconfig/puppetserver:
JAVA_ARGS="-Xms1g -Xmx1g -XX:MaxPermSize=1g" As requested, the puppetserver.service file:
$ cat /usr/lib/systemd/system/puppetserver.service [Unit] Description=puppetserver Service After=syslog.target network.target [Service] Type=simple EnvironmentFile=/etc/sysconfig/puppetserver User=puppet TimeoutStartSec=120 TimeoutStopSec=60 Restart=on-failure StartLimitBurst=5 PermissionsStartOnly=true ExecStartPre=/usr/bin/install --directory --owner=puppet --group=puppet --mode=775 /var/run/puppetlabs/puppetserver ExecStart=/usr/bin/java $JAVA_ARGS \ '-XX:OnOutOfMemoryError=kill -9 %%p' \ -Djava.security.egd=/dev/urandom \ -cp "${INSTALL_DIR}/puppet-server-release.jar" clojure.main \ -m puppetlabs.trapperkeeper.main \ --config "${CONFIG}" \ -b "${BOOTSTRAP_CONFIG}" $@ KillMode=process ExecStartPost=/bin/bash "${INSTALL_DIR}/ezbake-functions.sh" wait_for_app SuccessExitStatus=143 StandardOutput=syslog [Install] WantedBy=multi-user.target An attempt at running the ExecStartPost command by hand:
$ /usr/bin/java -Xms1g -Xmx1g -XX:MaxPermSize=1g -XX:OnOutOfMemoryError='kill -9 %%p' -Djava.security.egd=/dev/urandom -cp /opt/puppetlabs/server/apps/puppetserver/puppet-server-release.jar clojure.main -m puppetlabs.trapperkeeper.main --config /etc/puppetlabs/puppetserver/conf.d -b /etc/puppetlabs/puppetserver/bootstrap.cfg RuntimeError: Got 2 failure(s) while initializing: File[/var/log/puppetlabs/puppetserver]: change from 0700 to 0750 failed: failed to set mode 0700 on /var/log/puppetlabs/puppetserver: Operation not permitted - No message available; File[/var/run/puppetlabs/puppetserver]: change from 0775 to 0755 failed: failed to set mode 0775 on /var/run/puppetlabs/puppetserver: Operation not permitted - No message available So I tried again, but this time I changed some directory permissions, but still similar error (which doesn't make sense given I just changed the mode?):
$ sudo chown -R vagrant:vagrant /var/run/puppetlabs/ $ sudo chown -R vagrant:vagrant /var/log/puppetlabs/ $ sudo chmod -R 0755 /var/run/puppetlabs/ $ /usr/bin/java -Xms1g -Xmx1g -XX:MaxPermSize=1g -XX:OnOutOfMemoryError='kill -9 %%p' -Djava.security.egd=/dev/urandom -cp /opt/puppetlabs/server/apps/puppetserver/puppet-server-release.jar clojure.main -m puppetlabs.trapperkeeper.main --config /etc/puppetlabs/puppetserver/conf.d -b /etc/puppetlabs/puppetserver/bootstrap.cfg OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=1g; support was removed in 8.0 RuntimeError: Got 1 failure(s) while initializing: File[/var/run/puppetlabs/puppetserver]: change from 0775 to 0755 failed: failed to set mode 0775 on /var/run/puppetlabs/puppetserver: Operation not permitted - No message available use at /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/settings.rb:1007 What could be the issue?
3 Answers
Answers 1
Are you certain that it’s an OnOutOfMemory error? I ask because I found that the latest PuppetServer includes a newer version of logback, as shown by this message in /var/log/messages:
Mar 18 01:56:21 puppetserver java: Exception in thread "main" java.lang.AbstractMethodError: ch.qos.logback.core.net.SyslogAppenderBase.createOutputStream()Lch/qos/logback/core/net/SyslogOutputStream; Mar 18 01:56:21 puppetserver java: at ch.qos.logback.core.net.SyslogAppenderBase.start(SyslogAppenderBase.java:62) Mar 18 01:56:21 puppetserver java: at ch.qos.logback.classic.net.SyslogAppender.start(SyslogAppender.java:48) If you see, this replace “classic.net.Syslog” in logback.xml with "core.net.Syslog"
sed -i_old -e 's/classic.net.Syslog/core.net.Syslog/' /etc/puppetlabs/puppetserver/logback.xml If that's not the problem, please post your logfiles.
Answers 2
You have provided log as
OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=1g; support was removed in 8.0
So, it is clear that you are using jdk 8 which removed permgen space.
The Permanent Generation (PermGen) space has completely been removed and is kind of replaced by a new space called Metaspace. The consequences of the PermGen removal is that obviously the PermSize and MaxPermSize JVM arguments are ignored and you will never get a java.lang.OutOfMemoryError: PermGen error.
- So please remove the
-XX:MaxPermSize=1gportion fromJAVA_ARGSof location/etc/sysconfig/puppetserver
JAVA_ARGS="-Xms1g -Xmx1g"
So
- And then,
your command will be look like below:
$ /usr/bin/java -Xms1g -Xmx1g -XX:OnOutOfMemoryError='kill -9 %%p' -Djava.security.egd=/dev/urandom -cp /opt/puppetlabs/server/apps/puppetserver/puppet-server-release.jar clojure.main -m puppetlabs.trapperkeeper.main --config /etc/puppetlabs/puppetserver/conf.d -b /etc/puppetlabs/puppetserver/bootstrap.cfg or
$ /usr/bin/java -Xms1g -Xmx1g -Djava.security.egd=/dev/urandom -cp /opt/puppetlabs/server/apps/puppetserver/puppet-server-release.jar clojure.main -m puppetlabs.trapperkeeper.main --config /etc/puppetlabs/puppetserver/conf.d -b /etc/puppetlabs/puppetserver/bootstrap.cfg Please try this 2 commands. I hope both will run successfully. For caution purpose, I have added both.
N.B: There is no need to change the file permission. keep them as before you using.
Otherwise, if you don't want to change anything, you can downgrade java 8 to java 7
Related Link:
Answers 3
@lollercoster, you're starting your service with:
sudo puppet resource service puppetserver ensure=running But your next command is not telling which user it is
/usr/bin/java -Xms1g -Xmx1g -XX:MaxPermSize=1g -XX:OnOutOfMemoryError='kill -9 %%p' -Djava.security.egd=/dev/urandom -cp /opt/puppetlabs/server/apps/puppetserver/puppet-server-release.jar clojure.main -m puppetlabs.trapperkeeper.main --config /etc/puppetlabs/puppetserver/conf.d -b /etc/puppetlabs/puppetserver/bootstrap.cfg Please run whoami directly before your run the above command:
whoami /usr/bin/java -Xms1g -Xmx1g -XX:MaxPermSize=1g -XX:OnOutOfMemoryError='kill -9 %%p' -Djava.security.egd=/dev/urandom -cp /opt/puppetlabs/server/apps/puppetserver/puppet-server-release.jar clojure.main -m puppetlabs.trapperkeeper.main --config /etc/puppetlabs/puppetserver/conf.d -b /etc/puppetlabs/puppetserver/bootstrap.cfg Why this commands? I am pretty confident that your issue is a permission/usercontext issue and following commands make it more worse:
$ sudo chown -R vagrant:vagrant /var/run/puppetlabs/ $ sudo chown -R vagrant:vagrant /var/log/puppetlabs/ $ sudo chmod -R 0755 /var/run/puppetlabs/ with the above cases you have to do a sudo on the right user context to have the files writable, but it depends if the other data is writable. So you can give it a try, but I would use the puppet user created for the service
User=puppet So your puppet service runs as user puppet, but your logfiles are under vagrant user control, and not writable for puppet user.
So I would also try to switch back to:
$ sudo chown -R puppet /var/run/puppetlabs/ $ sudo chown -R puppet /var/log/puppetlabs/ $ sudo chmod -R 0755 /var/run/puppetlabs/ Java
Also I suggest you NOT to mess around with the JVM heap parameters, except you know what you do. Since you do not need to enforce a lower and a max bound on Java VM heap since JDK5. I would only limit to the upper bound
-Xmx1g The heap and the real allocated mem will be managed well by the javaVM. As JVM needs more it will step by step increase the heap and keep the needed size (no downsizing to a lower value will occur). So you will have a better used RAM on your machine.
Also please switch to the Oracle JVM. I am facing often issues with security and Compatibility with OpenJDK. So my first test is to run it on the latest needed JDK of Oracle. In your case Oracle JDK8. But please try first the permission things I mentioned before.
Good luck, and please keep me updated.
0 comments:
Post a Comment