Kibana is not starting up properly. When I open up the console it appears to be a javascript resource issue. When I open the js files directly (clicking on their link in the console) it appears they are incomplete and have been abruptly been cut off. Not sure if this is a browser file limit or somehow my files have been cut off? Please see the images below to show you what Im seeing.
File as seen in chrome. This is the very bottom of the file as per how chrome loads it.
I have restarted kibana to see if that would resolve it, no luck.
I think browsers have a max line limit in js files. I am not sure why kibana hasn't minified the js files? has it started up in some dev mode?
question summary
I guess I have discovered the reason for kibana not loading is because of the js not fully loading, this would change my question to how can I get all of my javascript to load?
Update
I have located the JS files in the kibana bundles folder and found that the file is fully intact. It is indeed a browser loading complete file issue. I'm confused why suddenly those files are too long to be loaded by the browser? Was working fine a fortnight ago. Still trying to work out how I can get chrome to load the files.
As suggested by @asettouf I have removed(backed up) bundles folder in the /opt/kibana/optimize directory and started kibana up again. This did re-generate the bundles folder but the files are identical, meaning I still have the same issue. How come Kibana is not minifying the js when it bundles the files for caching?
My kibana.yml. I think it is cleaner to paste a link to it:
went back turned on verbose logging and this is my output from deleting optimize folder and restarting. nothing stands out as an error message to me.
/var/log/kibana/kibana.log
replaced hostname with localhost for privacy and security reasons
I think this is an error with the webpack module not compiling the JS correctly. however i dont know enough about the module to debug it.
the files in question in the optimize folder are:
commons.bundle.js which is 65723 lines
kibana.bundle.js at 108950 lines
These are far from optimized and the content inside the files are not minified.
Result of curl -v localhost:5601
CURL REQUEST DIRECTLY TO KIBANA JS RESOURCES
I can confirm that curl -ukibanaadmin -v http://localhost/bundles/commons.bundle.js returns me the full 108950 lined JS file and curl -ukibanaadmin -v http://actual_kibana_hostname/bundles/commons.bundle.js returns the same content (number of lines) as my browser.
With that information I can make the assumption that this is not a Kibana issue at all. As mentioned by @val it is probably a setting in nginx that is preventing the entire file from being served unless accessed by localhost.
I think I can rule out Cloudflare as I still get the issue when I hit my server directly from the browser.
Thanks to everyone's suggestions and help so far. Getting closer and closer to solving my issue. I will do some more research on Nginx and its settings.
NGINX UPDATE
Nginx appears to only be loading the first 72kbs of my havascript files. Ive search all around my nginx config files and cannot see this setting anywhere. Ive added things like
location / { ... proxy_max_temp_file_size 1m; ... } and
location / { ... sendfile on; sendfile_max_chunk 1m; ... } But still I'm unable to overwrite this weird setting that is only allowing first part of file being loaded.
The connection uses nginx proxy_pass to foward port 80 to kibanas port '5601'. I feel like there could be a setting that limits file transfer over proxy? just not sure where to find it.
proxypass connection looks like:
server { listen 80; server_name logs.mydomain.com; auth_basic "Restricted Access"; auth_basic_user_file /etc/nginx/htpasswd.users; location / { proxy_pass http://localhost:5601; proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection 'upgrade'; proxy_set_header Host $host; proxy_cache_bypass $http_upgrade; } } And my default nginx settings is posted here.
/etc/nginx/nginx.conf
I have tried adding sendfile_max_chunk 512k and changed worker_processes from 4 to 2. Any other config things were already there. I was not the the one who initially set up the ELK stack.
And after the mentioned changes it looks like this:
/etc/nginx/nginx.conf
1 Answers
Answers 1
Since you're accessing Kibana behind a proxy, you'd need to bypass the proxy and see if the problem persists, i.e. compare the load times of
// through proxy curl -v kibana_host/bundles/commons.bundle.js curl -v kibana_host/bundles/kibana.bundle.js // bypass proxy curl -v localhost:5601/bundles/commons.bundle.js curl -v localhost:5601/bundles/kibana.bundle.js If the load time is lower when you bypass the proxy then you know it's not a Kibana issue but more with the way your proxy is configured.
UPDATE
Since we've narrowed this down to a proxy issue, try to update your nginx config with this
sendfile_max_chunk 512k worker_processes 2 UPDATE 2
Try another modification of your nginx configuration:
proxy_buffering: off; 
0 comments:
Post a Comment