Quantcast
Viewing all articles
Browse latest Browse all 2

Why are files missing in the first Docker exec but available in the second?

This is a very weird situation. We run an alpine-based docker container on Travis CI. Then we run ls as follows:

docker exec $CONTAINER_NAME ls -la specific/directory
docker exec $CONTAINER_NAME ls -la specific/directory

The first command shows a list of files but missing some files which should be there (part of the built image and does exist inside the image when investigated on other platforms).

Example output: Please notice the number in file_#

total 108
-rw-rw-r--    1 nginx    nginx         2069 Oct 21 06:08 2018_01_10_101010_file_1_here.php
-rw-rw-r--    1 nginx    nginx         1586 Oct 21 06:08 2018_01_12_101010_file_3_here.php
-rw-rw-r--    1 nginx    nginx         1649 Oct 21 06:08 2018_01_13_101010_file_4_here.php
-rw-rw-r--    1 nginx    nginx         1490 Oct 21 06:08 2018_01_14_101010_file_5_here.php
-rw-rw-r--    1 nginx    nginx          946 Oct 21 06:08 2018_01_15_101010_file_6_here.php
-rw-rw-r--    1 nginx    nginx         2379 Oct 21 06:08 2018_01_16_101010_file_7_here.php

The second command shows a full list of files as expected.

total 192
drwxrwxr-x    1 nginx    nginx         4096 Oct 21 06:08 .
drwxrwxr-x    1 nginx    nginx         4096 Oct 21 06:08 ..
-rw-rw-r--    1 nginx    nginx         2069 Oct 21 06:08 2018_01_10_101010_file_1_here.php
-rw-rw-r--    1 nginx    nginx         1758 Oct 21 06:08 2018_01_11_101010_file_2_here.php
-rw-rw-r--    1 nginx    nginx         1586 Oct 21 06:08 2018_01_12_101010_file_3_here.php
-rw-rw-r--    1 nginx    nginx         1649 Oct 21 06:08 2018_01_13_101010_file_4_here.php
-rw-rw-r--    1 nginx    nginx         1490 Oct 21 06:08 2018_01_14_101010_file_5_here.php
-rw-rw-r--    1 nginx    nginx          946 Oct 21 06:08 2018_01_15_101010_file_6_here.php
-rw-rw-r--    1 nginx    nginx         2379 Oct 21 06:08 2018_01_16_101010_file_7_here.php
-rw-rw-r--    1 nginx    nginx         1583 Oct 21 06:08 2018_01_17_101010_file_8_here.php
-rw-rw-r--    1 nginx    nginx          920 Oct 21 06:08 2018_01_18_101010_file_9_here.php

I also notice that the first command does not show the . and .. directories.

Why does that happen? And how to avoid it?

BTW: It does not only happen with ls. When the application code tries to list the files for the first time it shows the reduced list. But listing them again shows the full list!


The following are more debuging commands asked by @DanilaKiver and @ClemensKaserer

Output of docker info:

Containers: 4
 Running: 4
 Paused: 0
 Stopped: 0
Images: 30
Server Version: 18.06.0-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: d64c661f1d51c48782c9cec8fda7604785f93587
runc version: 69663f0bd4b60df09991c08812a60108003fa340
init version: fec3683
Security Options:
 seccomp
  Profile: default
Kernel Version: 4.15.0-1028-gcp
Operating System: Ubuntu 16.04.6 LTS
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 7.298GiB
Name: travis-job-8895eb3f-fd49-4241-af47-47daddf7beb5
ID: AVP5:DERI:ZI6V:ZTOR:KAAL:SPUS:ZQHT:NOOK:6IFE:VCFZ:CHL4:3QAK
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Registry Mirrors:
 https://mirror.gcr.io/
Live Restore Enabled: false

Running the following:

dmesg | grep ovl
docker exec $CONTAINER_NAME sync & ls -la specific/directory
docker exec $CONTAINER_NAME ls -la specific/directory
docker exec $CONTAINER_NAME ls -la specific/directory
dmesg | grep ovl
docker exec $CONTAINER_NAME dmesg | grep ovl

Gives:

# dmesg | grep ovl
[  178.665709] WARNING: CPU: 1 PID: 8854 at /build/linux-gcp-yd4gNt/linux-gcp-4.15.0/fs/overlayfs/readdir.c:417 ovl_iterate+0x25d/0x270 [overlay]
[  178.665761] RIP: 0010:ovl_iterate+0x25d/0x270 [overlay]
[  178.730507] WARNING: CPU: 0 PID: 9214 at /build/linux-gcp-yd4gNt/linux-gcp-4.15.0/fs/overlayfs/readdir.c:417 ovl_iterate+0x25d/0x270 [overlay]
[  178.730554] RIP: 0010:ovl_iterate+0x25d/0x270 [overlay]
[  178.777656] WARNING: CPU: 0 PID: 9214 at /build/linux-gcp-yd4gNt/linux-gcp-4.15.0/fs/overlayfs/readdir.c:417 ovl_iterate+0x25d/0x270 [overlay]
[  178.777707] RIP: 0010:ovl_iterate+0x25d/0x270 [overlay]

# docker exec $CONTAINER_NAME sync & ls -la specific/directory
total 184
drwxrwxr-x 2 travis travis 4096 Oct 21 11:38 .
drwxrwxr-x 5 travis travis 4096 Oct 21 11:38 ..
-rw-rw-r-- 1 travis travis 1156 Oct 21 11:38 2018_04_18_000001_file_01_here.php
-rw-rw-r-- 1 travis travis 1758 Oct 21 11:38 2018_04_18_000001_file_02_here.php
-rw-rw-r-- 1 travis travis 1598 Oct 21 11:38 2018_04_18_000001_file_03_here.php
-rw-rw-r-- 1 travis travis 1661 Oct 21 11:38 2018_04_18_000001_file_04_here.php
-rw-rw-r-- 1 travis travis 1502 Oct 21 11:38 2018_04_18_000001_file_05_here.php
-rw-rw-r-- 1 travis travis  958 Oct 21 11:38 2018_04_18_000001_file_06_here.php
-rw-rw-r-- 1 travis travis 2391 Oct 21 11:38 2018_04_18_000001_file_07_here.php
-rw-rw-r-- 1 travis travis 1583 Oct 21 11:38 2018_04_18_000001_file_08_here.php
-rw-rw-r-- 1 travis travis  932 Oct 21 11:38 2018_04_18_000001_file_09_here.php
-rw-rw-r-- 1 travis travis 1947 Oct 21 11:38 2018_04_18_000001_file_10_here.php
-rw-rw-r-- 1 travis travis 1484 Oct 21 11:38 2018_04_18_000001_file_11_here.php
-rw-rw-r-- 1 travis travis 1026 Oct 21 11:38 2018_04_18_000001_file_12_here.php
-rw-rw-r-- 1 travis travis 2170 Oct 21 11:38 2018_04_18_000001_file_13_here.php
-rw-rw-r-- 1 travis travis 1683 Oct 21 11:38 2018_04_18_000001_file_14_here.php
-rw-rw-r-- 1 travis travis  999 Oct 21 11:38 2018_04_18_000001_file_15_here.php
-rw-rw-r-- 1 travis travis 1477 Oct 21 11:38 2018_04_18_000001_file_16_here.php
-rw-rw-r-- 1 travis travis 1687 Oct 21 11:38 2018_04_18_000001_file_17_here.php
-rw-rw-r-- 1 travis travis 1619 Oct 21 11:38 2018_04_18_000001_file_18_here.php
-rw-rw-r-- 1 travis travis 1954 Oct 21 11:38 2018_04_18_000001_file_19_here.php
-rw-rw-r-- 1 travis travis 1203 Oct 21 11:38 2018_04_18_000001_file_20_here.php
-rw-rw-r-- 1 travis travis 1024 Oct 21 11:38 2018_04_18_000001_file_21_here.php
-rw-rw-r-- 1 travis travis 1290 Oct 21 11:38 2018_04_18_000001_file_22_here.php
-rw-rw-r-- 1 travis travis 1675 Oct 21 11:38 2018_04_18_000001_file_23_here.php
-rw-rw-r-- 1 travis travis 1411 Oct 21 11:38 2018_04_18_000001_file_24_here.php
-rw-rw-r-- 1 travis travis 1566 Oct 21 11:38 2018_04_18_000001_file_25_here.php
-rw-rw-r-- 1 travis travis  958 Oct 21 11:38 2018_04_18_000001_file_26_here.php
-rw-rw-r-- 1 travis travis 1122 Oct 21 11:38 2018_04_18_000001_file_27_here.php
-rw-rw-r-- 1 travis travis 1131 Oct 21 11:38 2018_04_18_000001_file_28_here.php
-rw-rw-r-- 1 travis travis 3034 Oct 21 11:38 2018_04_18_000001_file_29_here.php
-rw-rw-r-- 1 travis travis 1866 Oct 21 11:38 2018_04_18_000001_file_30_here.php
-rw-rw-r-- 1 travis travis 2068 Oct 21 11:38 2018_04_18_000001_file_31_here.php
-rw-rw-r-- 1 travis travis  991 Oct 21 11:38 2018_04_18_000001_file_32_here.php
-rw-rw-r-- 1 travis travis  995 Oct 21 11:38 2018_04_18_000001_file_33_here.php
-rw-rw-r-- 1 travis travis  933 Oct 21 11:38 2018_04_18_000001_file_34_here.php
-rw-rw-r-- 1 travis travis 2238 Oct 21 11:38 2018_04_18_000001_file_35_here.php
-rw-rw-r-- 1 travis travis  943 Oct 21 11:38 2018_04_18_000001_file_36_here.php
-rw-rw-r-- 1 travis travis 1765 Oct 21 11:38 2018_04_18_000001_file_37_here.php
-rw-rw-r-- 1 travis travis 2895 Oct 21 11:38 2018_04_18_000001_file_38_here.php
-rw-rw-r-- 1 travis travis 1501 Oct 21 11:38 2018_04_18_000001_file_39_here.php
-rw-rw-r-- 1 travis travis 1444 Oct 21 11:38 2018_04_18_000001_file_40_here.php
-rw-rw-r-- 1 travis travis 1722 Oct 21 11:38 2018_04_18_000001_file_41_here.php
-rw-rw-r-- 1 travis travis 1404 Oct 21 11:38 2018_04_18_000001_file_42_here.php
-rw-rw-r-- 1 travis travis  941 Oct 21 11:38 2018_04_18_000001_file_43_here.php
-rw-rw-r-- 1 travis travis  919 Oct 21 11:38 2018_04_18_000001_file_44_here.php

# docker exec $CONTAINER_NAME ls -la specific/directory
total 140
-rw-rw-r--    1 nginx    nginx         1758 Oct 21 11:38 2018_04_18_000001_file_02_here.php
-rw-rw-r--    1 nginx    nginx         1598 Oct 21 11:38 2018_04_18_000001_file_03_here.php
-rw-rw-r--    1 nginx    nginx         1661 Oct 21 11:38 2018_04_18_000001_file_04_here.php
-rw-rw-r--    1 nginx    nginx         1502 Oct 21 11:38 2018_04_18_000001_file_05_here.php
-rw-rw-r--    1 nginx    nginx          958 Oct 21 11:38 2018_04_18_000001_file_06_here.php
-rw-rw-r--    1 nginx    nginx         2391 Oct 21 11:38 2018_04_18_000001_file_07_here.php
-rw-rw-r--    1 nginx    nginx         1583 Oct 21 11:38 2018_04_18_000001_file_08_here.php
-rw-rw-r--    1 nginx    nginx         1484 Oct 21 11:38 2018_04_18_000001_file_11_here.php
-rw-rw-r--    1 nginx    nginx         1026 Oct 21 11:38 2018_04_18_000001_file_12_here.php
-rw-rw-r--    1 nginx    nginx         2170 Oct 21 11:38 2018_04_18_000001_file_13_here.php
-rw-rw-r--    1 nginx    nginx         1683 Oct 21 11:38 2018_04_18_000001_file_14_here.php
-rw-rw-r--    1 nginx    nginx         1477 Oct 21 11:38 2018_04_18_000001_file_16_here.php
-rw-rw-r--    1 nginx    nginx         1687 Oct 21 11:38 2018_04_18_000001_file_17_here.php
-rw-rw-r--    1 nginx    nginx         1954 Oct 21 11:38 2018_04_18_000001_file_19_here.php
-rw-rw-r--    1 nginx    nginx         1203 Oct 21 11:38 2018_04_18_000001_file_20_here.php
-rw-rw-r--    1 nginx    nginx         1024 Oct 21 11:38 2018_04_18_000001_file_21_here.php
-rw-rw-r--    1 nginx    nginx         1290 Oct 21 11:38 2018_04_18_000001_file_22_here.php
-rw-rw-r--    1 nginx    nginx         1675 Oct 21 11:38 2018_04_18_000001_file_23_here.php
-rw-rw-r--    1 nginx    nginx         1411 Oct 21 11:38 2018_04_18_000001_file_24_here.php
-rw-rw-r--    1 nginx    nginx         1566 Oct 21 11:38 2018_04_18_000001_file_25_here.php
-rw-rw-r--    1 nginx    nginx          958 Oct 21 11:38 2018_04_18_000001_file_26_here.php
-rw-rw-r--    1 nginx    nginx         1122 Oct 21 11:38 2018_04_18_000001_file_27_here.php
-rw-rw-r--    1 nginx    nginx         1131 Oct 21 11:38 2018_04_18_000001_file_28_here.php
-rw-rw-r--    1 nginx    nginx         2068 Oct 21 11:38 2018_04_18_000001_file_31_here.php
-rw-rw-r--    1 nginx    nginx          991 Oct 21 11:38 2018_04_18_000001_file_32_here.php
-rw-rw-r--    1 nginx    nginx          995 Oct 21 11:38 2018_04_18_000001_file_33_here.php
-rw-rw-r--    1 nginx    nginx          933 Oct 21 11:38 2018_04_18_000001_file_34_here.php
-rw-rw-r--    1 nginx    nginx         2238 Oct 21 11:38 2018_04_18_000001_file_35_here.php
-rw-rw-r--    1 nginx    nginx          943 Oct 21 11:38 2018_04_18_000001_file_36_here.php
-rw-rw-r--    1 nginx    nginx         2895 Oct 21 11:38 2018_04_18_000001_file_38_here.php
-rw-rw-r--    1 nginx    nginx         1501 Oct 21 11:38 2018_04_18_000001_file_39_here.php
-rw-rw-r--    1 nginx    nginx         1444 Oct 21 11:38 2018_04_18_000001_file_40_here.php
-rw-rw-r--    1 nginx    nginx         1722 Oct 21 11:38 2018_04_18_000001_file_41_here.php
-rw-rw-r--    1 nginx    nginx         1404 Oct 21 11:38 2018_04_18_000001_file_42_here.php
-rw-rw-r--    1 nginx    nginx          941 Oct 21 11:38 2018_04_18_000001_file_43_here.php

# docker exec $CONTAINER_NAME ls -la specific/directory
total 192
drwxrwxr-x    1 nginx    nginx         4096 Oct 21 11:38 .
drwxrwxr-x    1 nginx    nginx         4096 Oct 21 11:38 ..
-rw-rw-r--    1 nginx    nginx         1156 Oct 21 11:38 2018_04_18_000001_file_01_here.php
-rw-rw-r--    1 nginx    nginx         1758 Oct 21 11:38 2018_04_18_000001_file_02_here.php
-rw-rw-r--    1 nginx    nginx         1598 Oct 21 11:38 2018_04_18_000001_file_03_here.php
-rw-rw-r--    1 nginx    nginx         1661 Oct 21 11:38 2018_04_18_000001_file_04_here.php
-rw-rw-r--    1 nginx    nginx         1502 Oct 21 11:38 2018_04_18_000001_file_05_here.php
-rw-rw-r--    1 nginx    nginx          958 Oct 21 11:38 2018_04_18_000001_file_06_here.php
-rw-rw-r--    1 nginx    nginx         2391 Oct 21 11:38 2018_04_18_000001_file_07_here.php
-rw-rw-r--    1 nginx    nginx         1583 Oct 21 11:38 2018_04_18_000001_file_08_here.php
-rw-rw-r--    1 nginx    nginx          932 Oct 21 11:38 2018_04_18_000001_file_09_here.php
-rw-rw-r--    1 nginx    nginx         1947 Oct 21 11:38 2018_04_18_000001_file_10_here.php
-rw-rw-r--    1 nginx    nginx         1484 Oct 21 11:38 2018_04_18_000001_file_11_here.php
-rw-rw-r--    1 nginx    nginx         1026 Oct 21 11:38 2018_04_18_000001_file_12_here.php
-rw-rw-r--    1 nginx    nginx         2170 Oct 21 11:38 2018_04_18_000001_file_13_here.php
-rw-rw-r--    1 nginx    nginx         1683 Oct 21 11:38 2018_04_18_000001_file_14_here.php
-rw-rw-r--    1 nginx    nginx          999 Oct 21 11:38 2018_04_18_000001_file_15_here.php
-rw-rw-r--    1 nginx    nginx         1477 Oct 21 11:38 2018_04_18_000001_file_16_here.php
-rw-rw-r--    1 nginx    nginx         1687 Oct 21 11:38 2018_04_18_000001_file_17_here.php
-rw-rw-r--    1 nginx    nginx         1619 Oct 21 11:38 2018_04_18_000001_file_18_here.php
-rw-rw-r--    1 nginx    nginx         1954 Oct 21 11:38 2018_04_18_000001_file_19_here.php
-rw-rw-r--    1 nginx    nginx         1203 Oct 21 11:38 2018_04_18_000001_file_20_here.php
-rw-rw-r--    1 nginx    nginx         1024 Oct 21 11:38 2018_04_18_000001_file_21_here.php
-rw-rw-r--    1 nginx    nginx         1290 Oct 21 11:38 2018_04_18_000001_file_22_here.php
-rw-rw-r--    1 nginx    nginx         1675 Oct 21 11:38 2018_04_18_000001_file_23_here.php
-rw-rw-r--    1 nginx    nginx         1411 Oct 21 11:38 2018_04_18_000001_file_24_here.php
-rw-rw-r--    1 nginx    nginx         1566 Oct 21 11:38 2018_04_18_000001_file_25_here.php
-rw-rw-r--    1 nginx    nginx          958 Oct 21 11:38 2018_04_18_000001_file_26_here.php
-rw-rw-r--    1 nginx    nginx         1122 Oct 21 11:38 2018_04_18_000001_file_27_here.php
-rw-rw-r--    1 nginx    nginx         1131 Oct 21 11:38 2018_04_18_000001_file_28_here.php
-rw-rw-r--    1 nginx    nginx         3034 Oct 21 11:38 2018_04_18_000001_file_29_here.php
-rw-rw-r--    1 nginx    nginx         1866 Oct 21 11:38 2018_04_18_000001_file_30_here.php
-rw-rw-r--    1 nginx    nginx         2068 Oct 21 11:38 2018_04_18_000001_file_31_here.php
-rw-rw-r--    1 nginx    nginx          991 Oct 21 11:38 2018_04_18_000001_file_32_here.php
-rw-rw-r--    1 nginx    nginx          995 Oct 21 11:38 2018_04_18_000001_file_33_here.php
-rw-rw-r--    1 nginx    nginx          933 Oct 21 11:38 2018_04_18_000001_file_34_here.php
-rw-rw-r--    1 nginx    nginx         2238 Oct 21 11:38 2018_04_18_000001_file_35_here.php
-rw-rw-r--    1 nginx    nginx          943 Oct 21 11:38 2018_04_18_000001_file_36_here.php
-rw-rw-r--    1 nginx    nginx         1765 Oct 21 11:38 2018_04_18_000001_file_37_here.php
-rw-rw-r--    1 nginx    nginx         2895 Oct 21 11:38 2018_04_18_000001_file_38_here.php
-rw-rw-r--    1 nginx    nginx         1501 Oct 21 11:38 2018_04_18_000001_file_39_here.php
-rw-rw-r--    1 nginx    nginx         1444 Oct 21 11:38 2018_04_18_000001_file_40_here.php
-rw-rw-r--    1 nginx    nginx         1722 Oct 21 11:38 2018_04_18_000001_file_41_here.php
-rw-rw-r--    1 nginx    nginx         1404 Oct 21 11:38 2018_04_18_000001_file_42_here.php
-rw-rw-r--    1 nginx    nginx          941 Oct 21 11:38 2018_04_18_000001_file_43_here.php
-rw-rw-r--    1 nginx    nginx          919 Oct 21 11:38 2018_04_18_000001_file_44_here.php

# dmesg | grep ovl
[  178.665709] WARNING: CPU: 1 PID: 8854 at /build/linux-gcp-yd4gNt/linux-gcp-4.15.0/fs/overlayfs/readdir.c:417 ovl_iterate+0x25d/0x270 [overlay]
[  178.665761] RIP: 0010:ovl_iterate+0x25d/0x270 [overlay]
[  178.730507] WARNING: CPU: 0 PID: 9214 at /build/linux-gcp-yd4gNt/linux-gcp-4.15.0/fs/overlayfs/readdir.c:417 ovl_iterate+0x25d/0x270 [overlay]
[  178.730554] RIP: 0010:ovl_iterate+0x25d/0x270 [overlay]
[  178.777656] WARNING: CPU: 0 PID: 9214 at /build/linux-gcp-yd4gNt/linux-gcp-4.15.0/fs/overlayfs/readdir.c:417 ovl_iterate+0x25d/0x270 [overlay]
[  178.777707] RIP: 0010:ovl_iterate+0x25d/0x270 [overlay]
[  213.441060] WARNING: CPU: 1 PID: 9378 at /build/linux-gcp-yd4gNt/linux-gcp-4.15.0/fs/overlayfs/readdir.c:417 ovl_iterate+0x25d/0x270 [overlay]
[  213.441107] RIP: 0010:ovl_iterate+0x25d/0x270 [overlay]

# docker exec $CONTAINER_NAME dmesg | grep ovl
[  178.665709] WARNING: CPU: 1 PID: 8854 at /build/linux-gcp-yd4gNt/linux-gcp-4.15.0/fs/overlayfs/readdir.c:417 ovl_iterate+0x25d/0x270 [overlay]
[  178.665761] RIP: 0010:ovl_iterate+0x25d/0x270 [overlay]
[  178.730507] WARNING: CPU: 0 PID: 9214 at /build/linux-gcp-yd4gNt/linux-gcp-4.15.0/fs/overlayfs/readdir.c:417 ovl_iterate+0x25d/0x270 [overlay]
[  178.730554] RIP: 0010:ovl_iterate+0x25d/0x270 [overlay]
[  178.777656] WARNING: CPU: 0 PID: 9214 at /build/linux-gcp-yd4gNt/linux-gcp-4.15.0/fs/overlayfs/readdir.c:417 ovl_iterate+0x25d/0x270 [overlay]
[  178.777707] RIP: 0010:ovl_iterate+0x25d/0x270 [overlay]
[  213.441060] WARNING: CPU: 1 PID: 9378 at /build/linux-gcp-yd4gNt/linux-gcp-4.15.0/fs/overlayfs/readdir.c:417 ovl_iterate+0x25d/0x270 [overlay]
[  213.441107] RIP: 0010:ovl_iterate+0x25d/0x270 [overlay]

Updated:

This problem happens on Ubuntu Xenial 16.04 but it does not happen on Ubuntu Bionic 18.04. So, it seems to be a bug in the kernel and got solved in later versions.


Viewing all articles
Browse latest Browse all 2

Trending Articles



<script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>