Bug 1399659

Summary: fetch build artifacts by build is fetching for all
Product: [Internal] Red Hat Internal Copr Reporter: Eduardo <ecerqueira>
Component: backendAssignee: Pavel Raiskup <praiskup>
Status: CLOSED CURRENTRELEASE QA Contact:
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: unspecifiedCC: clime, jmazanek, praiskup
Target Milestone: ---   
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-09-18 09:16:37 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Eduardo 2016-11-29 14:17:37 UTC
Description of problem:


Version-Release number of selected component (if applicable): copr-cli-1.57-1.fc24.noarch


How reproducible:

Steps to Reproduce:
1. from your copr frontend get a build number that has artifacts ( logs, rpm, srpm, etc ) in my cases 6205 
2. from your machine where copr-cli is installed call download-build as following command:

copr-cli --config /home/ecerquei/.config/copr-ecerquei download-build --dest /home/ecerquei/tmp/copr 6205


Actual results:

I got all artifacts from all builds

Expected results:

artifacts from only build id specified.

Additional info:


[ecerquei@dev copr]$ copr-cli --config /home/ecerquei/.config/copr-ecerquei download-build --help
usage: copr
download-build [-h] [-r CHROOTS] [--dest DEST] build_id

positional arguments:
  build_id              Build ID

optional arguments:
  -h, --help            show this help message and exit
  -r CHROOTS, --chroot CHROOTS
                        Select chroots to fetch
  --dest DEST, -d DEST  Base directory to store packages

I ran:

copr-cli --config /home/ecerquei/.config/copr-ecerquei download-build --dest /home/ecerquei/tmp/copr *6205*

and it is what I got:

.
├── epel-7-x86_64
│   ├── 00006205-paws
│   ├── 128x128
│   │   ├── index.html
│   │   ├── index.html?C=D;O=A
│   │   ├── index.html?C=M;O=A
│   │   ├── index.html?C=N;O=D
│   │   └── index.html?C=S;O=A
│   ├──
46903ba8f362190d8abcdd15bffb89bd57c2ecdb11f4fc72e7ea05a88578dad4-primary.xml.gz
│   ├── 64x64
│   │   ├── index.html
│   │   ├── index.html?C=D;O=A
│   │   ├── index.html?C=M;O=A
│   │   ├── index.html?C=N;O=D
│   │   └── index.html?C=S;O=A
│   ├──
7d4ec0e1c902ee6f4f9fa5da8924854248662cab189509f42a9a7393ffed3e27-filelists.xml.gz
│   ├── appstream-failed.xml.gz
│   ├── appstream-ignore.xml.gz
│   ├── appstream.xml.gz
│   ├──
bf16629ad35a761be8ff1f3cbb05ef22b2a41731c73ac214a94a3133cbb7b77e-filelists.sqlite.bz2
│   ├── build-00005927.log
│   ├── build-00006082.log
│   ├── build-00006082.rsync.log
│   ├── build-00006083.log
│   ├── build-00006083.rsync.log
│   ├── build-00006085.log
│   ├── build-00006085.rsync.log
│   ├── build-00006086.log
│   ├── build-00006086.rsync.log
│   ├── build-00006087.log
│   ├── build-00006087.rsync.log
│   ├── build-00006088.log
│   ├── build-00006088.rsync.log
│   ├── build-00006122.log
│   ├── build-00006122.rsync.log
│   ├── build-00006133.log
│   ├── build-00006133.rsync.log
│   ├── build-00006143.log
│   ├── build-00006143.rsync.log
│   ├── build-00006144.log
│   ├── build-00006144.rsync.log
│   ├── build-00006186.log
│   ├── build-00006186.rsync.log
│   ├── build-00006205.log
│   ├── build-00006205.rsync.log
│   ├── build.info
│   ├── build.log.gz
│   ├──
cd4bbd8ee1fa0e2543682e75c6bb4d66089dd16255a07156955d194acd5603bc-other.xml.gz
│   ├──
eb3c98d20d4d73f2347805656863128770b412a70f59602243804fc428b056c3-other.sqlite.bz2
│   ├── fail
│   ├──
ff188778d585bcfd10886a26f5506685d14252ec6de224158b39e6c8264c50ac-primary.sqlite.bz2
│   ├── index.html
│   ├── index.html?C=D;O=A
│   ├── index.html?C=D;O=D
│   ├── index.html?C=M;O=A
│   ├── index.html?C=M;O=D
│   ├── index.html?C=N;O=A
│   ├── index.html?C=N;O=D
│   ├── index.html?C=S;O=A
│   ├── index.html?C=S;O=D
│   ├── mockchain.log.gz
│   ├── paws-0.3.3-0.noarch.rpm
│   ├── paws-0.3.3-0.src.rpm
│   ├── paws-0.3.3-0.x86_64.rpm
│   ├── paws-0.3.4-0.noarch.rpm
│   ├── paws-0.3.4-0.src.rpm
│   ├── repomd.xml
│   ├── root.log.gz
│   ├── state.log.gz
│   └── success
├── fedora-24-x86_64
│   ├── 00006205-paws
│   ├──
069599c0b9372dca4afa2cb05ec9f1190a27f0f4ad63cc134ca51ad48dcd6124-primary.xml.gz
│   ├── 128x128
│   │   ├── index.html
│   │   ├── index.html?C=D;O=A
│   │   ├── index.html?C=M;O=A
│   │   ├── index.html?C=N;O=D
│   │   └── index.html?C=S;O=A
│   ├──
3366f60a921f877c41a5612a454733b6f75977ffb24d42bc60b49038b34bb435-filelists.xml.gz
│   ├──
3551b0f0c85db6e4259e05eff310edfc8b3ce30b9070de7dd996080748809e64-primary.sqlite.bz2
│   ├── 64x64
│   │   ├── index.html
│   │   ├── index.html?C=D;O=A
│   │   ├── index.html?C=M;O=A
│   │   ├── index.html?C=N;O=D
│   │   └── index.html?C=S;O=A
│   ├──
73d08097e9c967ab21cee6d5c752980e11777eb1c80bae876703866a47384bde-other.sqlite.bz2
│   ├──
95f3673428bbde0741a274886ca2e18e8432d5ec01a486a4e30eb071576d50a4-other.xml.gz
│   ├── appstream-failed.xml.gz
│   ├── appstream-ignore.xml.gz
│   ├── appstream.xml.gz
│   ├── build-00005927.log
│   ├── build-00005927.rsync.log
│   ├── build-00006082.log
│   ├── build-00006082.rsync.log
│   ├── build-00006083.log
│   ├── build-00006083.rsync.log
│   ├── build-00006085.log
│   ├── build-00006085.rsync.log
│   ├── build-00006086.log
│   ├── build-00006086.rsync.log
│   ├── build-00006087.log
│   ├── build-00006087.rsync.log
│   ├── build-00006088.log
│   ├── build-00006088.rsync.log
│   ├── build-00006122.log
│   ├── build-00006122.rsync.log
│   ├── build-00006133.log
│   ├── build-00006133.rsync.log
│   ├── build-00006142.log
│   ├── build-00006142.rsync.log
│   ├── build-00006143.log
│   ├── build-00006143.rsync.log
│   ├── build-00006144.log
│   ├── build-00006144.rsync.log
│   ├── build-00006186.log
│   ├── build-00006186.rsync.log
│   ├── build-00006205.log
│   ├── build-00006205.rsync.log
│   ├── build.info
│   ├── build.log.gz
│   ├──
f46ed1ada6e2e2f8fc9ff567fb90491f561def45064625b8469747b5397162e9-filelists.sqlite.bz2
│   ├── fail
│   ├── index.html
│   ├── index.html?C=D;O=A
│   ├── index.html?C=D;O=D
│   ├── index.html?C=M;O=A
│   ├── index.html?C=M;O=D
│   ├── index.html?C=N;O=A
│   ├── index.html?C=N;O=D
│   ├── index.html?C=S;O=A
│   ├── index.html?C=S;O=D
│   ├── mockchain.log.gz
│   ├── paws-0.3.3-0.noarch.rpm
│   ├── paws-0.3.3-0.src.rpm
│   ├── paws-0.3.3-0.x86_64.rpm
│   ├── paws-0.3.4-0.noarch.rpm
│   ├── paws-0.3.4-0.src.rpm
│   ├── repomd.xml
│   ├── root.log.gz
│   ├── state.log.gz
│   └── success
└── rhel-7-x86_64
    ├── 00006205-paws
    ├── 128x128
    │   ├── index.html
    │   ├── index.html?C=D;O=A
    │   ├── index.html?C=M;O=A
    │   ├── index.html?C=N;O=D
    │   └── index.html?C=S;O=A
    ├──
34fdbefd1dbacbbdcbfe918cac495980275d536979ddfc48a35beb9ad6699bbb-primary.sqlite.bz2
    ├──
5345eb976d9f6ed6355e8781127476b4b9448747350397ded99100e556bdf30e-other.xml.gz
    ├── 64x64
    │   ├── index.html
    │   ├── index.html?C=D;O=A
    │   ├── index.html?C=M;O=A
    │   ├── index.html?C=N;O=D
    │   └── index.html?C=S;O=A
    ├──
711f7276ab8523672fa278e12262002ff77d48d7e863c16df976a5df9708e574-primary.xml.gz
    ├──
a5a6ac281c81c36f4e5f864c78b3a930ca6405fd2dc9b63f8c825147f06c3dc5-filelists.sqlite.bz2
    ├── appstream-failed.xml.gz
    ├── appstream-ignore.xml.gz
    ├── appstream.xml.gz
    ├──
bdced3847f5dcdccc07f3369360c47fbd71083ed9817283509ff174732cfc5fb-other.sqlite.bz2
    ├── build-00005927.log
    ├── build-00005927.rsync.log
    ├── build-00006082.log
    ├── build-00006082.rsync.log
    ├── build-00006083.log
    ├── build-00006083.rsync.log
    ├── build-00006085.log
    ├── build-00006085.rsync.log
    ├── build-00006086.log
    ├── build-00006086.rsync.log
    ├── build-00006087.log
    ├── build-00006087.rsync.log
    ├── build-00006088.log
    ├── build-00006088.rsync.log
    ├── build-00006122.log
    ├── build-00006122.rsync.log
    ├── build-00006133.log
    ├── build-00006133.rsync.log
    ├── build-00006143.log
    ├── build-00006143.rsync.log
    ├── build-00006144.log
    ├── build-00006144.rsync.log
    ├── build-00006186.log
    ├── build-00006186.rsync.log
    ├── build-00006205.log
    ├── build-00006205.rsync.log
    ├── build.info
    ├── build.log.gz
    ├── fail
    ├──
fe1daa32d12815107c6399682cd2d8e0a4edf2c8c768addd4b38504ed6e405a4-filelists.xml.gz
    ├── index.html
    ├── index.html?C=D;O=A
    ├── index.html?C=D;O=D
    ├── index.html?C=M;O=A
    ├── index.html?C=M;O=D
    ├── index.html?C=N;O=A
    ├── index.html?C=N;O=D
    ├── index.html?C=S;O=A
    ├── index.html?C=S;O=D
    ├── mockchain.log.gz
    ├── paws-0.3.3-0.noarch.rpm
    ├── paws-0.3.3-0.src.rpm
    ├── paws-0.3.3-0.x86_64.rpm
    ├── paws-0.3.4-0.noarch.rpm
    ├── paws-0.3.4-0.src.rpm
    ├── repomd.xml
    ├── root.log.gz
    ├── state.log.gz
    └── success

9 directories, 205 files

Comment 1 Pavel Raiskup 2016-11-29 14:27:09 UTC
Thanks for the report!  As we talked in email, this is a bug most probably
fixed upstream and it waits for upcoming internal copr update (WIP, should
happen soon).

I'll make sure 'download-build' works as expected.

Comment 2 Pavel Raiskup 2016-12-12 15:11:34 UTC
This seems to be upstream problem, too.  Switching against upstream.

Seems like the 'download-build' action is implemented as simple recursive
`wget` call:
https://pagure.io/copr/copr/blob/06ff349145704bbd8637345f807a582bfaaf1736/f/cli/copr_cli/main.py#_371

This needs a bit discussion probably.  The issues are:

- we should drop the 'results' column entirely
- we changed layout of build directory in history several times, so somebody
  (most probably Backend) should keep the list of files related to particular
  build in separate file, then client could download the file-list and
  download only appropriate files

As I'm not sure what is this feature for, I would also consider dropping the
download-build functionality entirely (as long as there is reasonable feature
available in dnf).

Comment 3 clime 2018-04-09 07:18:41 UTC
Hello, I've tried with copr-cli-1.67 to download a build from a fedora copr instance:

$ mkdir /tmp/foo && cd /tmp/foo
$ copr-cli download-build 738638

It created subdirectory for each chroot (fedora-27-x86_64, epel-7-x86_64, ...) and inside it put result files for each respective builddir (i.e. into fedora-27-x86_64 subdir, it downloaded this build content: https://copr-be.cloud.fedoraproject.org/results/kanarip/blesk/fedora-27-x86_64/00738638-suricata/).

So this is exactly how the feature should work. I am not sure now if the problem was there and disappeared or if it was a deployment-specific problem, or ...

Comment 4 clime 2018-04-09 07:20:47 UTC
@praiskup, @Eduardo: could you, please, try again with copr-cli-1.67 ideally against both internal instance and Fedora COPR instance?

Comment 5 clime 2018-04-09 09:32:49 UTC
Also copr-cli-1.66 can be used.

Comment 6 Pavel Raiskup 2018-04-09 11:29:43 UTC
Right, this seems to work fine for Fedora Copr ATM.

Comment 7 Pavel Raiskup 2018-09-18 09:16:37 UTC
After the upgrade of internal copr, this feature seems to work as expected.
Please reopen in if you disagree, and thanks for the report!