Tuesday, April 24, 2018

Issue with akka application inside docker container

By Hường Hana 12:30 PM , , Leave a Comment

We have a akka application running in production in a non clustered mode ( doesn't akka clustering ). We are trying to dockerize this application and running into an issue. We are using docker host networking and for various other reasons we are not yet ready to use other types of networking.

Application binds to 0.0.0.0 interface on port 9080 and does a simple http healthcheck on itself. This healthcheck is failing because we are seeing a connection reset error. We are seeing this only inside docker container.

During startup time application gets bound to 0:0:0:0:0:0:0:0:9080. 2018-04-13T08:29:54.921-0700 [INFO ] - Bound to /0:0:0:0:0:0:0:0:9080

akka logs:

2018-04-13T08:34:25.311-0700 [DEBUG] akka.io.TcpListener            [15:34:25.311UTC] - New connection accepted 2018-04-13T08:34:25.312-0700 [DEBUG] akka.io.SelectionHandler       [15:34:25.312UTC] - now supervising Actor[akka://appname/system/IO-TCP/selectors/$d/12#394551476] 2018-04-13T08:34:25.312-0700 [DEBUG] akka.io.TcpIncomingConnection  [15:34:25.312UTC] - started (akka.io.TcpIncomingConnection@1dafecc7) 2018-04-13T08:34:25.312-0700 [DEBUG] akka.io.TcpIncomingConnection  [15:34:25.312UTC] - now watched by Actor[akka://appname/system/IO-TCP/selectors/$d#992175932] 2018-04-13T08:34:25.312-0700 [DEBUG] akka.io.TcpIncomingConnection  [15:34:25.312UTC] - now watched by Actor[akka://appname/user/IO-HTTP/listener-0/88#-1187840999] 2018-04-13T08:34:25.313-0700 [DEBUG] akka.io.SelectionHandler       [15:34:25.313UTC] - now supervising Actor[akka://appname/system/IO-TCP/selectors/$h/11#789402148] 2018-04-13T08:34:25.313-0700 [DEBUG] akka.io.TcpOutgoingConnection  [15:34:25.313UTC] - started (akka.io.TcpOutgoingConnection@62c77639) 2018-04-13T08:34:25.313-0700 [DEBUG] akka.io.TcpOutgoingConnection  [15:34:25.313UTC] - now watched by Actor[akka://appname/system/IO-TCP/selectors/$h#227381172] 2018-04-13T08:34:25.313-0700 [DEBUG] akka.io.TcpOutgoingConnection  [15:34:25.313UTC] - Attempting connection to [/0.0.0.0:9080] 2018-04-13T08:34:25.313-0700 [DEBUG] akka.io.TcpListener            [15:34:25.313UTC] - New connection accepted 2018-04-13T08:34:25.313-0700 [DEBUG] akka.io.TcpOutgoingConnection  [15:34:25.313UTC] - Connection established to [/0.0.0.0:9080] 2018-04-13T08:34:25.314-0700 [DEBUG] akka.io.TcpOutgoingConnection  [15:34:25.314UTC] - now watched by Actor[akka://appname/user/IO-HTTP/group-0/52#1518564728] 2018-04-13T08:34:25.314-0700 [DEBUG] akka.io.SelectionHandler       [15:34:25.314UTC] - now supervising Actor[akka://appname/system/IO-TCP/selectors/$h/12#1641348988] 2018-04-13T08:34:25.314-0700 [DEBUG] akka.io.TcpIncomingConnection  [15:34:25.314UTC] - started (akka.io.TcpIncomingConnection@65bac45) 2018-04-13T08:34:25.314-0700 [DEBUG] akka.io.TcpIncomingConnection  [15:34:25.314UTC] - now watched by Actor[akka://appname/system/IO-TCP/selectors/$h#227381172] 2018-04-13T08:34:26.327-0700 [DEBUG] akka.io.TcpIncomingConnection  [15:34:26.327UTC] - now watched by Actor[akka://appname/user/IO-HTTP/listener-0/89#-1079090421] 2018-04-13T08:34:26.327-0700 [DEBUG] akka.io.TcpIncomingConnection  [15:34:26.327UTC] - stopped 2018-04-13T08:34:26.327-0700 [DEBUG] akka.io.TcpOutgoingConnection  [15:34:26.327UTC] - Closing connection due to IO error java.io.IOException: Connection reset by peer 2018-04-13T08:34:26.327-0700 [DEBUG] akka.io.TcpOutgoingConnection  [15:34:26.327UTC] - stopped 2018-04-13T08:34:26.328-0700 [DEBUG] akka.io.TcpIncomingConnection  [15:34:26.328UTC] - stopped

Spray logs:

2018-04-13T08:35:39.168-0700 - Dispatching GET request to http://10.4.4.178:9080/admin/health to handler Actor[akka://appname/system/IO-TCP/selectors/$b/20#-595408603] 2018-04-13T08:35:45.296-0700 - Dispatching GET request to http://10.4.4.178:9080/admin/health to handler Actor[akka://appname/system/IO-TCP/selectors/$d/19#721242322] 2018-04-13T08:35:51.372-0700 - Dispatching GET request to http://10.4.4.178:9080/admin/health to handler Actor[akka://appname/system/IO-TCP/selectors/$c/16#1510321611] 2018-04-13T08:35:57.436-0700 - Dispatching GET request to http://10.4.4.178:9080/admin/health to handler Actor[akka://appname/system/IO-TCP/selectors/$g/22#1994752981] 2018-04-13T08:36:03.622-0700 - Dispatching GET request to http://10.4.4.178:9080/admin/health to handler Actor[akka://appname/system/IO-TCP/selectors/$f/20#-1450294853]

Healthcheck logs

2018-04-13T08:32:53.356-0700 [INFO ] - [BEGIN] Health check 2018-04-13T08:32:53.356-0700 [DEBUG] - Memory    : SUCCESS 2018-04-13T08:32:54.378-0700 [DEBUG] - HTTP      : FAILURE, spray.can.Http$ConnectionException: ErrorClosed(Connection reset by peer) 2018-04-13T08:32:54.378-0700 [ERROR] - spray.can.Http$ConnectionException: ErrorClosed(Connection reset by peer)     at spray.can.client.HttpHostConnectionSlot.reportDisconnection(HttpHostConnectionSlot.scala:228)     at spray.can.client.HttpHostConnectionSlot$$anonfun$connected$1.applyOrElse(HttpHostConnectionSlot.scala:161)     at akka.actor.Actor$class.aroundReceive(Actor.scala:465)     at spray.can.client.HttpHostConnectionSlot.aroundReceive(HttpHostConnectionSlot.scala:33)     at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)     at akka.actor.ActorCell.invoke_aroundBody0(ActorCell.scala:487)     at akka.actor.ActorCell$AjcClosure1.run(ActorCell.scala:1)     at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)     at akka.instrumentation.ActorCellInstrumentation.aroundBehaviourInvoke(ActorCellInstrumentation.scala:66)     at akka.actor.ActorCell.invoke(ActorCell.scala:483)     at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)     at akka.dispatch.Mailbox.run(Mailbox.scala:220)     at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393)     at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)     at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)     at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)     at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)  2018-04-13T08:32:54.378-0700 [INFO ] - [END  ] Status: Failed (1022 ms)

Dockerfile:

FROM java-base:0.2.3 EXPOSE 9080 9081 ENV MAIN_CLASS="akka.kernel.Main" \     ADMIN_PORT="9080" \     HEALTHCHECK_URI="/admin/health" \     TEMPLATES="/app/templates/override.settings.tmpl:/app/conf/override.settings /app/templates/environment.conf.tmpl:/app/conf/environment.conf" \     JMX_PORT="9081" \     APP_NAME="appname" \     APP_VERSION="1.34.0" \     CLASSPATH="/app/conf" \     ENV_FILES="/app/conf/override.settings" \     INSTRUMENTATION_PATH="/app/ext" \     JVM_MISC_OPTS="-Dpid=1 \                   -Dlog4j.configuration=file:///app/conf/log4j.xml \                   -javaagent:\${INSTRUMENTATION_PATH}/aspectjweaver-1.7.4.jar \                   -Dlog4j.configuration=file:///app/conf/log4j.xml \                   -Dlogs.dir=/app/logs \                   -Dakka.home=/app \                   -Dakka.kernel.quiet=false \                   -Dconfig.file=/app/conf/environment.conf" \     MAIN_CLASS_ARGS=appname.Boot ADD ext /app/ext ADD deps /app/libs ADD templates/ /app/templates/

Docker-compose file:

version: '3.3' services:   appname:     container_name: appname     image: appname:1.34.0     network_mode: host     restart: unless-stopped     ulimits:       nofile:         soft: 65535         hard: 65535     environment:       UID:       SDLC_ENVIRONMENT:       REGION:       PARTITION:       TAGS:       INTERNAL_IP:       CONSUL_ADDRESS:       DOCKER_GID:       MAIN_CLASS_ARGS: appname.Boot     volumes:       - "./logs:/app/logs"

Versions: akka - 2.3.0 spray - 1.3.2 running on alpine linux inside a docker container.

Any pointers would help. Thanks in advance.

1 Answers

Answers 1

In the docker-compose file, define port mapping under appname.

ports:     9080:9080
If You Enjoyed This, Take 5 Seconds To Share It

0 comments:

Post a Comment