This is a very weird situation. We run an alpine-based docker container on Travis CI. Then we run ls
as follows:
docker exec $CONTAINER_NAME ls -la specific/directory
docker exec $CONTAINER_NAME ls -la specific/directory
The first command shows a list of files but missing some files which should be there (part of the built image and does exist inside the image when investigated on other platforms).
Example output: Please notice the number in file_#
total 108
-rw-rw-r-- 1 nginx nginx 2069 Oct 21 06:08 2018_01_10_101010_file_1_here.php
-rw-rw-r-- 1 nginx nginx 1586 Oct 21 06:08 2018_01_12_101010_file_3_here.php
-rw-rw-r-- 1 nginx nginx 1649 Oct 21 06:08 2018_01_13_101010_file_4_here.php
-rw-rw-r-- 1 nginx nginx 1490 Oct 21 06:08 2018_01_14_101010_file_5_here.php
-rw-rw-r-- 1 nginx nginx 946 Oct 21 06:08 2018_01_15_101010_file_6_here.php
-rw-rw-r-- 1 nginx nginx 2379 Oct 21 06:08 2018_01_16_101010_file_7_here.php
The second command shows a full list of files as expected.
total 192
drwxrwxr-x 1 nginx nginx 4096 Oct 21 06:08 .
drwxrwxr-x 1 nginx nginx 4096 Oct 21 06:08 ..
-rw-rw-r-- 1 nginx nginx 2069 Oct 21 06:08 2018_01_10_101010_file_1_here.php
-rw-rw-r-- 1 nginx nginx 1758 Oct 21 06:08 2018_01_11_101010_file_2_here.php
-rw-rw-r-- 1 nginx nginx 1586 Oct 21 06:08 2018_01_12_101010_file_3_here.php
-rw-rw-r-- 1 nginx nginx 1649 Oct 21 06:08 2018_01_13_101010_file_4_here.php
-rw-rw-r-- 1 nginx nginx 1490 Oct 21 06:08 2018_01_14_101010_file_5_here.php
-rw-rw-r-- 1 nginx nginx 946 Oct 21 06:08 2018_01_15_101010_file_6_here.php
-rw-rw-r-- 1 nginx nginx 2379 Oct 21 06:08 2018_01_16_101010_file_7_here.php
-rw-rw-r-- 1 nginx nginx 1583 Oct 21 06:08 2018_01_17_101010_file_8_here.php
-rw-rw-r-- 1 nginx nginx 920 Oct 21 06:08 2018_01_18_101010_file_9_here.php
I also notice that the first command does not show the .
and ..
directories.
Why does that happen? And how to avoid it?
BTW: It does not only happen with ls
. When the application code tries to list the files for the first time it shows the reduced list. But listing them again shows the full list!
The following are more debuging commands asked by @DanilaKiver and @ClemensKaserer
Output of docker info
:
Containers: 4
Running: 4
Paused: 0
Stopped: 0
Images: 30
Server Version: 18.06.0-ce
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: d64c661f1d51c48782c9cec8fda7604785f93587
runc version: 69663f0bd4b60df09991c08812a60108003fa340
init version: fec3683
Security Options:
seccomp
Profile: default
Kernel Version: 4.15.0-1028-gcp
Operating System: Ubuntu 16.04.6 LTS
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 7.298GiB
Name: travis-job-8895eb3f-fd49-4241-af47-47daddf7beb5
ID: AVP5:DERI:ZI6V:ZTOR:KAAL:SPUS:ZQHT:NOOK:6IFE:VCFZ:CHL4:3QAK
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Registry Mirrors:
https://mirror.gcr.io/
Live Restore Enabled: false
Running the following:
dmesg | grep ovl
docker exec $CONTAINER_NAME sync & ls -la specific/directory
docker exec $CONTAINER_NAME ls -la specific/directory
docker exec $CONTAINER_NAME ls -la specific/directory
dmesg | grep ovl
docker exec $CONTAINER_NAME dmesg | grep ovl
Gives:
# dmesg | grep ovl
[ 178.665709] WARNING: CPU: 1 PID: 8854 at /build/linux-gcp-yd4gNt/linux-gcp-4.15.0/fs/overlayfs/readdir.c:417 ovl_iterate+0x25d/0x270 [overlay]
[ 178.665761] RIP: 0010:ovl_iterate+0x25d/0x270 [overlay]
[ 178.730507] WARNING: CPU: 0 PID: 9214 at /build/linux-gcp-yd4gNt/linux-gcp-4.15.0/fs/overlayfs/readdir.c:417 ovl_iterate+0x25d/0x270 [overlay]
[ 178.730554] RIP: 0010:ovl_iterate+0x25d/0x270 [overlay]
[ 178.777656] WARNING: CPU: 0 PID: 9214 at /build/linux-gcp-yd4gNt/linux-gcp-4.15.0/fs/overlayfs/readdir.c:417 ovl_iterate+0x25d/0x270 [overlay]
[ 178.777707] RIP: 0010:ovl_iterate+0x25d/0x270 [overlay]
# docker exec $CONTAINER_NAME sync & ls -la specific/directory
total 184
drwxrwxr-x 2 travis travis 4096 Oct 21 11:38 .
drwxrwxr-x 5 travis travis 4096 Oct 21 11:38 ..
-rw-rw-r-- 1 travis travis 1156 Oct 21 11:38 2018_04_18_000001_file_01_here.php
-rw-rw-r-- 1 travis travis 1758 Oct 21 11:38 2018_04_18_000001_file_02_here.php
-rw-rw-r-- 1 travis travis 1598 Oct 21 11:38 2018_04_18_000001_file_03_here.php
-rw-rw-r-- 1 travis travis 1661 Oct 21 11:38 2018_04_18_000001_file_04_here.php
-rw-rw-r-- 1 travis travis 1502 Oct 21 11:38 2018_04_18_000001_file_05_here.php
-rw-rw-r-- 1 travis travis 958 Oct 21 11:38 2018_04_18_000001_file_06_here.php
-rw-rw-r-- 1 travis travis 2391 Oct 21 11:38 2018_04_18_000001_file_07_here.php
-rw-rw-r-- 1 travis travis 1583 Oct 21 11:38 2018_04_18_000001_file_08_here.php
-rw-rw-r-- 1 travis travis 932 Oct 21 11:38 2018_04_18_000001_file_09_here.php
-rw-rw-r-- 1 travis travis 1947 Oct 21 11:38 2018_04_18_000001_file_10_here.php
-rw-rw-r-- 1 travis travis 1484 Oct 21 11:38 2018_04_18_000001_file_11_here.php
-rw-rw-r-- 1 travis travis 1026 Oct 21 11:38 2018_04_18_000001_file_12_here.php
-rw-rw-r-- 1 travis travis 2170 Oct 21 11:38 2018_04_18_000001_file_13_here.php
-rw-rw-r-- 1 travis travis 1683 Oct 21 11:38 2018_04_18_000001_file_14_here.php
-rw-rw-r-- 1 travis travis 999 Oct 21 11:38 2018_04_18_000001_file_15_here.php
-rw-rw-r-- 1 travis travis 1477 Oct 21 11:38 2018_04_18_000001_file_16_here.php
-rw-rw-r-- 1 travis travis 1687 Oct 21 11:38 2018_04_18_000001_file_17_here.php
-rw-rw-r-- 1 travis travis 1619 Oct 21 11:38 2018_04_18_000001_file_18_here.php
-rw-rw-r-- 1 travis travis 1954 Oct 21 11:38 2018_04_18_000001_file_19_here.php
-rw-rw-r-- 1 travis travis 1203 Oct 21 11:38 2018_04_18_000001_file_20_here.php
-rw-rw-r-- 1 travis travis 1024 Oct 21 11:38 2018_04_18_000001_file_21_here.php
-rw-rw-r-- 1 travis travis 1290 Oct 21 11:38 2018_04_18_000001_file_22_here.php
-rw-rw-r-- 1 travis travis 1675 Oct 21 11:38 2018_04_18_000001_file_23_here.php
-rw-rw-r-- 1 travis travis 1411 Oct 21 11:38 2018_04_18_000001_file_24_here.php
-rw-rw-r-- 1 travis travis 1566 Oct 21 11:38 2018_04_18_000001_file_25_here.php
-rw-rw-r-- 1 travis travis 958 Oct 21 11:38 2018_04_18_000001_file_26_here.php
-rw-rw-r-- 1 travis travis 1122 Oct 21 11:38 2018_04_18_000001_file_27_here.php
-rw-rw-r-- 1 travis travis 1131 Oct 21 11:38 2018_04_18_000001_file_28_here.php
-rw-rw-r-- 1 travis travis 3034 Oct 21 11:38 2018_04_18_000001_file_29_here.php
-rw-rw-r-- 1 travis travis 1866 Oct 21 11:38 2018_04_18_000001_file_30_here.php
-rw-rw-r-- 1 travis travis 2068 Oct 21 11:38 2018_04_18_000001_file_31_here.php
-rw-rw-r-- 1 travis travis 991 Oct 21 11:38 2018_04_18_000001_file_32_here.php
-rw-rw-r-- 1 travis travis 995 Oct 21 11:38 2018_04_18_000001_file_33_here.php
-rw-rw-r-- 1 travis travis 933 Oct 21 11:38 2018_04_18_000001_file_34_here.php
-rw-rw-r-- 1 travis travis 2238 Oct 21 11:38 2018_04_18_000001_file_35_here.php
-rw-rw-r-- 1 travis travis 943 Oct 21 11:38 2018_04_18_000001_file_36_here.php
-rw-rw-r-- 1 travis travis 1765 Oct 21 11:38 2018_04_18_000001_file_37_here.php
-rw-rw-r-- 1 travis travis 2895 Oct 21 11:38 2018_04_18_000001_file_38_here.php
-rw-rw-r-- 1 travis travis 1501 Oct 21 11:38 2018_04_18_000001_file_39_here.php
-rw-rw-r-- 1 travis travis 1444 Oct 21 11:38 2018_04_18_000001_file_40_here.php
-rw-rw-r-- 1 travis travis 1722 Oct 21 11:38 2018_04_18_000001_file_41_here.php
-rw-rw-r-- 1 travis travis 1404 Oct 21 11:38 2018_04_18_000001_file_42_here.php
-rw-rw-r-- 1 travis travis 941 Oct 21 11:38 2018_04_18_000001_file_43_here.php
-rw-rw-r-- 1 travis travis 919 Oct 21 11:38 2018_04_18_000001_file_44_here.php
# docker exec $CONTAINER_NAME ls -la specific/directory
total 140
-rw-rw-r-- 1 nginx nginx 1758 Oct 21 11:38 2018_04_18_000001_file_02_here.php
-rw-rw-r-- 1 nginx nginx 1598 Oct 21 11:38 2018_04_18_000001_file_03_here.php
-rw-rw-r-- 1 nginx nginx 1661 Oct 21 11:38 2018_04_18_000001_file_04_here.php
-rw-rw-r-- 1 nginx nginx 1502 Oct 21 11:38 2018_04_18_000001_file_05_here.php
-rw-rw-r-- 1 nginx nginx 958 Oct 21 11:38 2018_04_18_000001_file_06_here.php
-rw-rw-r-- 1 nginx nginx 2391 Oct 21 11:38 2018_04_18_000001_file_07_here.php
-rw-rw-r-- 1 nginx nginx 1583 Oct 21 11:38 2018_04_18_000001_file_08_here.php
-rw-rw-r-- 1 nginx nginx 1484 Oct 21 11:38 2018_04_18_000001_file_11_here.php
-rw-rw-r-- 1 nginx nginx 1026 Oct 21 11:38 2018_04_18_000001_file_12_here.php
-rw-rw-r-- 1 nginx nginx 2170 Oct 21 11:38 2018_04_18_000001_file_13_here.php
-rw-rw-r-- 1 nginx nginx 1683 Oct 21 11:38 2018_04_18_000001_file_14_here.php
-rw-rw-r-- 1 nginx nginx 1477 Oct 21 11:38 2018_04_18_000001_file_16_here.php
-rw-rw-r-- 1 nginx nginx 1687 Oct 21 11:38 2018_04_18_000001_file_17_here.php
-rw-rw-r-- 1 nginx nginx 1954 Oct 21 11:38 2018_04_18_000001_file_19_here.php
-rw-rw-r-- 1 nginx nginx 1203 Oct 21 11:38 2018_04_18_000001_file_20_here.php
-rw-rw-r-- 1 nginx nginx 1024 Oct 21 11:38 2018_04_18_000001_file_21_here.php
-rw-rw-r-- 1 nginx nginx 1290 Oct 21 11:38 2018_04_18_000001_file_22_here.php
-rw-rw-r-- 1 nginx nginx 1675 Oct 21 11:38 2018_04_18_000001_file_23_here.php
-rw-rw-r-- 1 nginx nginx 1411 Oct 21 11:38 2018_04_18_000001_file_24_here.php
-rw-rw-r-- 1 nginx nginx 1566 Oct 21 11:38 2018_04_18_000001_file_25_here.php
-rw-rw-r-- 1 nginx nginx 958 Oct 21 11:38 2018_04_18_000001_file_26_here.php
-rw-rw-r-- 1 nginx nginx 1122 Oct 21 11:38 2018_04_18_000001_file_27_here.php
-rw-rw-r-- 1 nginx nginx 1131 Oct 21 11:38 2018_04_18_000001_file_28_here.php
-rw-rw-r-- 1 nginx nginx 2068 Oct 21 11:38 2018_04_18_000001_file_31_here.php
-rw-rw-r-- 1 nginx nginx 991 Oct 21 11:38 2018_04_18_000001_file_32_here.php
-rw-rw-r-- 1 nginx nginx 995 Oct 21 11:38 2018_04_18_000001_file_33_here.php
-rw-rw-r-- 1 nginx nginx 933 Oct 21 11:38 2018_04_18_000001_file_34_here.php
-rw-rw-r-- 1 nginx nginx 2238 Oct 21 11:38 2018_04_18_000001_file_35_here.php
-rw-rw-r-- 1 nginx nginx 943 Oct 21 11:38 2018_04_18_000001_file_36_here.php
-rw-rw-r-- 1 nginx nginx 2895 Oct 21 11:38 2018_04_18_000001_file_38_here.php
-rw-rw-r-- 1 nginx nginx 1501 Oct 21 11:38 2018_04_18_000001_file_39_here.php
-rw-rw-r-- 1 nginx nginx 1444 Oct 21 11:38 2018_04_18_000001_file_40_here.php
-rw-rw-r-- 1 nginx nginx 1722 Oct 21 11:38 2018_04_18_000001_file_41_here.php
-rw-rw-r-- 1 nginx nginx 1404 Oct 21 11:38 2018_04_18_000001_file_42_here.php
-rw-rw-r-- 1 nginx nginx 941 Oct 21 11:38 2018_04_18_000001_file_43_here.php
# docker exec $CONTAINER_NAME ls -la specific/directory
total 192
drwxrwxr-x 1 nginx nginx 4096 Oct 21 11:38 .
drwxrwxr-x 1 nginx nginx 4096 Oct 21 11:38 ..
-rw-rw-r-- 1 nginx nginx 1156 Oct 21 11:38 2018_04_18_000001_file_01_here.php
-rw-rw-r-- 1 nginx nginx 1758 Oct 21 11:38 2018_04_18_000001_file_02_here.php
-rw-rw-r-- 1 nginx nginx 1598 Oct 21 11:38 2018_04_18_000001_file_03_here.php
-rw-rw-r-- 1 nginx nginx 1661 Oct 21 11:38 2018_04_18_000001_file_04_here.php
-rw-rw-r-- 1 nginx nginx 1502 Oct 21 11:38 2018_04_18_000001_file_05_here.php
-rw-rw-r-- 1 nginx nginx 958 Oct 21 11:38 2018_04_18_000001_file_06_here.php
-rw-rw-r-- 1 nginx nginx 2391 Oct 21 11:38 2018_04_18_000001_file_07_here.php
-rw-rw-r-- 1 nginx nginx 1583 Oct 21 11:38 2018_04_18_000001_file_08_here.php
-rw-rw-r-- 1 nginx nginx 932 Oct 21 11:38 2018_04_18_000001_file_09_here.php
-rw-rw-r-- 1 nginx nginx 1947 Oct 21 11:38 2018_04_18_000001_file_10_here.php
-rw-rw-r-- 1 nginx nginx 1484 Oct 21 11:38 2018_04_18_000001_file_11_here.php
-rw-rw-r-- 1 nginx nginx 1026 Oct 21 11:38 2018_04_18_000001_file_12_here.php
-rw-rw-r-- 1 nginx nginx 2170 Oct 21 11:38 2018_04_18_000001_file_13_here.php
-rw-rw-r-- 1 nginx nginx 1683 Oct 21 11:38 2018_04_18_000001_file_14_here.php
-rw-rw-r-- 1 nginx nginx 999 Oct 21 11:38 2018_04_18_000001_file_15_here.php
-rw-rw-r-- 1 nginx nginx 1477 Oct 21 11:38 2018_04_18_000001_file_16_here.php
-rw-rw-r-- 1 nginx nginx 1687 Oct 21 11:38 2018_04_18_000001_file_17_here.php
-rw-rw-r-- 1 nginx nginx 1619 Oct 21 11:38 2018_04_18_000001_file_18_here.php
-rw-rw-r-- 1 nginx nginx 1954 Oct 21 11:38 2018_04_18_000001_file_19_here.php
-rw-rw-r-- 1 nginx nginx 1203 Oct 21 11:38 2018_04_18_000001_file_20_here.php
-rw-rw-r-- 1 nginx nginx 1024 Oct 21 11:38 2018_04_18_000001_file_21_here.php
-rw-rw-r-- 1 nginx nginx 1290 Oct 21 11:38 2018_04_18_000001_file_22_here.php
-rw-rw-r-- 1 nginx nginx 1675 Oct 21 11:38 2018_04_18_000001_file_23_here.php
-rw-rw-r-- 1 nginx nginx 1411 Oct 21 11:38 2018_04_18_000001_file_24_here.php
-rw-rw-r-- 1 nginx nginx 1566 Oct 21 11:38 2018_04_18_000001_file_25_here.php
-rw-rw-r-- 1 nginx nginx 958 Oct 21 11:38 2018_04_18_000001_file_26_here.php
-rw-rw-r-- 1 nginx nginx 1122 Oct 21 11:38 2018_04_18_000001_file_27_here.php
-rw-rw-r-- 1 nginx nginx 1131 Oct 21 11:38 2018_04_18_000001_file_28_here.php
-rw-rw-r-- 1 nginx nginx 3034 Oct 21 11:38 2018_04_18_000001_file_29_here.php
-rw-rw-r-- 1 nginx nginx 1866 Oct 21 11:38 2018_04_18_000001_file_30_here.php
-rw-rw-r-- 1 nginx nginx 2068 Oct 21 11:38 2018_04_18_000001_file_31_here.php
-rw-rw-r-- 1 nginx nginx 991 Oct 21 11:38 2018_04_18_000001_file_32_here.php
-rw-rw-r-- 1 nginx nginx 995 Oct 21 11:38 2018_04_18_000001_file_33_here.php
-rw-rw-r-- 1 nginx nginx 933 Oct 21 11:38 2018_04_18_000001_file_34_here.php
-rw-rw-r-- 1 nginx nginx 2238 Oct 21 11:38 2018_04_18_000001_file_35_here.php
-rw-rw-r-- 1 nginx nginx 943 Oct 21 11:38 2018_04_18_000001_file_36_here.php
-rw-rw-r-- 1 nginx nginx 1765 Oct 21 11:38 2018_04_18_000001_file_37_here.php
-rw-rw-r-- 1 nginx nginx 2895 Oct 21 11:38 2018_04_18_000001_file_38_here.php
-rw-rw-r-- 1 nginx nginx 1501 Oct 21 11:38 2018_04_18_000001_file_39_here.php
-rw-rw-r-- 1 nginx nginx 1444 Oct 21 11:38 2018_04_18_000001_file_40_here.php
-rw-rw-r-- 1 nginx nginx 1722 Oct 21 11:38 2018_04_18_000001_file_41_here.php
-rw-rw-r-- 1 nginx nginx 1404 Oct 21 11:38 2018_04_18_000001_file_42_here.php
-rw-rw-r-- 1 nginx nginx 941 Oct 21 11:38 2018_04_18_000001_file_43_here.php
-rw-rw-r-- 1 nginx nginx 919 Oct 21 11:38 2018_04_18_000001_file_44_here.php
# dmesg | grep ovl
[ 178.665709] WARNING: CPU: 1 PID: 8854 at /build/linux-gcp-yd4gNt/linux-gcp-4.15.0/fs/overlayfs/readdir.c:417 ovl_iterate+0x25d/0x270 [overlay]
[ 178.665761] RIP: 0010:ovl_iterate+0x25d/0x270 [overlay]
[ 178.730507] WARNING: CPU: 0 PID: 9214 at /build/linux-gcp-yd4gNt/linux-gcp-4.15.0/fs/overlayfs/readdir.c:417 ovl_iterate+0x25d/0x270 [overlay]
[ 178.730554] RIP: 0010:ovl_iterate+0x25d/0x270 [overlay]
[ 178.777656] WARNING: CPU: 0 PID: 9214 at /build/linux-gcp-yd4gNt/linux-gcp-4.15.0/fs/overlayfs/readdir.c:417 ovl_iterate+0x25d/0x270 [overlay]
[ 178.777707] RIP: 0010:ovl_iterate+0x25d/0x270 [overlay]
[ 213.441060] WARNING: CPU: 1 PID: 9378 at /build/linux-gcp-yd4gNt/linux-gcp-4.15.0/fs/overlayfs/readdir.c:417 ovl_iterate+0x25d/0x270 [overlay]
[ 213.441107] RIP: 0010:ovl_iterate+0x25d/0x270 [overlay]
# docker exec $CONTAINER_NAME dmesg | grep ovl
[ 178.665709] WARNING: CPU: 1 PID: 8854 at /build/linux-gcp-yd4gNt/linux-gcp-4.15.0/fs/overlayfs/readdir.c:417 ovl_iterate+0x25d/0x270 [overlay]
[ 178.665761] RIP: 0010:ovl_iterate+0x25d/0x270 [overlay]
[ 178.730507] WARNING: CPU: 0 PID: 9214 at /build/linux-gcp-yd4gNt/linux-gcp-4.15.0/fs/overlayfs/readdir.c:417 ovl_iterate+0x25d/0x270 [overlay]
[ 178.730554] RIP: 0010:ovl_iterate+0x25d/0x270 [overlay]
[ 178.777656] WARNING: CPU: 0 PID: 9214 at /build/linux-gcp-yd4gNt/linux-gcp-4.15.0/fs/overlayfs/readdir.c:417 ovl_iterate+0x25d/0x270 [overlay]
[ 178.777707] RIP: 0010:ovl_iterate+0x25d/0x270 [overlay]
[ 213.441060] WARNING: CPU: 1 PID: 9378 at /build/linux-gcp-yd4gNt/linux-gcp-4.15.0/fs/overlayfs/readdir.c:417 ovl_iterate+0x25d/0x270 [overlay]
[ 213.441107] RIP: 0010:ovl_iterate+0x25d/0x270 [overlay]
Updated:
This problem happens on Ubuntu Xenial 16.04 but it does not happen on Ubuntu Bionic 18.04. So, it seems to be a bug in the kernel and got solved in later versions.