Bug 697833

Summary: Local sync progress report "error_details" should match same format as grinder
Product: [Retired] Pulp Reporter: John Matthews <jmatthew>
Component: z_otherAssignee: John Matthews <jmatthew>
Status: CLOSED CURRENTRELEASE QA Contact: Preethi Thomas <pthomas>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: unspecifiedKeywords: Triaged
Target Milestone: ---   
Target Release: Sprint 23   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2011-08-16 12:10:18 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 647488    

Description John Matthews 2011-04-19 11:54:10 UTC 
Description of problem:
There is some discrepancy between the format of errors from Local Sync vs Yum Syncs

Local Sync:
"error_details": [
    [{"item_type": "rpm", "fileName": "/vault/repos/vt_and_trees_errors/xen-devel-3.0.3-105.el5_5.2.i386.rpm"}, 
     {"error_type": "<type 'exceptions.OSError'>", 
      "traceback": ["Traceback (most recent call last):", "  File \"/shared/repo/pulp/src/pulp/server/api/repo_sync.py\", line 553, in _sync_rpms", "    self._process_rpm(pkg, dst_repo_dir)", "  File \"/shared/repo/pulp/src/pulp/server/api/repo_sync.py\", line 523, in _process_rpm", "    pkg_info = pulp.server.util.get_rpm_information(pkg)", "  File \"/shared/repo/pulp/src/pulp/server/util.py\", line 69, in get_rpm_information", "    file_descriptor_number = os.open(rpm_path, os.O_RDONLY)", "OSError: [Errno 13] Permission denied: '/vault/repos/vt_and_trees_errors/xen-devel-3.0.3-105.el5_5.2.i386.rpm'"], "error": "[Errno 13] Permission denied: '/vault/repos/vt_and_trees_errors/xen-devel-3.0.3-105.el5_5.2.i386.rpm'"}], [{"item_type": "rpm", "fileName": "/vault/repos/vt_and_trees_errors/Virtualization-de-DE-5.2-9.noarch.rpm"}, {"error_type": "<type 'exceptions.OSError'>", "traceback": ["Traceback (most recent call last):", "  File \"/shared/repo/pulp/src/pulp/server/api/repo_sync.py\", line 553, in _sync_rpms", "    self._process_rpm(pkg, dst_repo_dir)", "  File \"/shared/repo/pulp/src/pulp/server/api/repo_sync.py\", line 523, in _process_rpm", "    pkg_info = pulp.server.util.get_rpm_information(pkg)", "  File \"/shared/repo/pulp/src/pulp/server/util.py\", line 69, in get_rpm_information", "    file_descriptor_number = os.open(rpm_path, os.O_RDONLY)", "OSError: [Errno 13] Permission denied: '/vault/repos/vt_and_trees_errors/Virtualization-de-DE-5.2-9.noarch.rpm'"], 
      "error": "[Errno 13] Permission denied: '/vault/repos/vt_and_trees_errors/Virtualization-de-DE-5.2-9.noarch.rpm'"}], 

Yum Sync:

  "error_details": 
          [[{"checksumtype": "sha256", "checksum": "435d92e6c09248b501b8d2ae786f92ccfad69fab8b1bc774e2b66ff6c0d83979", "fileName": "pulp-dot-2.0-test-0.1.2-1.fc11.x86_64.rpm", "downloadurl": "http://jmatthews.fedorapeople.org/repo_with_bad_read_perms/pulp-dot-2.0-test-0.1.2-1.fc11.x86_64.rpm", "item_type": "rpm", "savepath": "/var/lib/pulp//repos/repo_with_bad_read_perms//", "pkgpath": "/var/lib/pulp//packages/435/pulp-dot-2.0-test/0.1.2/1.fc11/x86_64", "size": 2359}, 
            {"error_type": "error", 
             "error": "HTTP status code of 403 received for http://jmatthews.fedorapeople.org/repo_with_bad_read_perms/pulp-dot-2.0-test-0.1.2-1.fc11.x86_64.rpm"}]],
Comment 1 John Matthews 2011-04-19 12:10:07 UTC 
*** Bug 697840 has been marked as a duplicate of this bug. ***
Comment 2 John Matthews 2011-04-27 16:34:11 UTC 
Fixed with below commit in pulp:
http://git.fedorahosted.org/git/?p=pulp.git;a=commitdiff;h=8ce84505bb3ce98d43ed59399a30f957cb4ffcf9

Commit in grinder:
http://git.fedorahosted.org/git/?p=grinder.git;a=commitdiff;h=a9ceea0698a896c283cb7d711b8be61d73a631b4

QE:

To test this you will need to look at the json output of a local sync and a yum sync.  Both of those syncs will need an error.  Compare the returned json and look to see if the general structure of the json object is the same.
Comment 3 Jay Dobies 2011-04-27 20:11:36 UTC 
Fixed in Pulp 0.172, grinder 0.96.
Comment 4 Preethi Thomas 2011-04-29 17:23:53 UTC 
verified

[root@preethi 752]# rpm -q pulp
pulp-0.0.172-1.fc14.noarch


local

[root@preethi 752]#  curl -k -u admin:admin https://localhost/pulp/api/repositories/local/sync/45acf902-7286-11e0-9bbe-002564a85a58/ |json_reformat
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1843  100  1843    0     0  11599      0 --:--:-- --:--:-- --:--:-- 11664
{
  "scheduled_time": null,
  "exception": null,
  "status_path": "/pulp/api/repositories/local/sync/45acf902-7286-11e0-9bbe-002564a85a58/",
  "finish_time": "1304098185",
  "start_time": "1304098184",
  "traceback": null,
  "method_name": "_sync",
  "state": "finished",
  "result": true,
  "progress": {
    "status": "running",
    "size_total": 4204656,
    "num_download": 0,
    "items_left": 0,
    "items_total": 4,
    "step": "Finished",
    "items_remaining": 0,
    "num_success": 3,
    "item_name": null,
    "size_left": 0,
    "details": {
      "rpm": {
        "num_success": 3,
        "total_count": 4,
        "items_left": 0,
        "size_left": 0,
        "num_error": 1,
        "total_size_bytes": 4204656
      }
    },
    "error_details": [
      {
        "traceback": [
          "Traceback (most recent call last):",
          "  File \"/usr/lib/python2.7/site-packages/pulp/server/api/repo_sync.py\", line 649, in _sync_rpms",
          "    self._process_rpm(pkg, dst_repo_dir)",
          "  File \"/usr/lib/python2.7/site-packages/pulp/server/api/repo_sync.py\", line 578, in _process_rpm",
          "    pkg_info = pulp.server.util.get_rpm_information(pkg)",
          "  File \"/usr/lib/python2.7/site-packages/pulp/server/util.py\", line 93, in get_rpm_information",
          "    file_descriptor_number = os.open(rpm_path, os.O_RDONLY)",
          "OSError: [Errno 13] Permission denied: '/var/lib/pulp/repos/repos/pulp/pulp/demo_repos/test_bandwidth_repo/pulp-large_1mb_test-packageB-0.1.1-1.fc14.noarch.rpm'"
        ],
        "checksumtype": "",
        "checksum": "",
        "error_type": "<type 'exceptions.OSError'>",
        "fileName": "/var/lib/pulp/repos/repos/pulp/pulp/demo_repos/test_bandwidth_repo/pulp-large_1mb_test-packageB-0.1.1-1.fc14.noarch.rpm",
        "downloadurl": "",
        "item_type": "",
        "savepath": "",
        "error": "[Errno 13] Permission denied: '/var/lib/pulp/repos/repos/pulp/pulp/demo_repos/test_bandwidth_repo/pulp-large_1mb_test-packageB-0.1.1-1.fc14.noarch.rpm'",
        "pkgpath": "",
        "size": ""
      }
    ],
    "num_error": 1
  },
  "id": "45acf902-7286-11e0-9bbe-002564a85a58"
}
  yum 

[root@preethi 752]#  curl -k -u admin:admin https://localhost/pulp/api/repositories/bad-repo/sync/2b36f5f0-7286-11e0-ae54-002564a85a58/ |json_reformat  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  2629    0  2629    0     0  15885      0 --:--:-- --:--:-- --:--:-- 16030
{
  "scheduled_time": null,
  "exception": null,
  "status_path": "/pulp/api/repositories/bad-repo/sync/2b36f5f0-7286-11e0-ae54-002564a85a58/",
  "finish_time": "1304098142",
  "start_time": "1304098139",
  "traceback": null,
  "method_name": "_sync",
  "state": "finished",
  "result": true,
  "progress": {
    "status": "FINISHED",
    "num_success": 0,
    "size_total": 6791,
    "num_download": 0,
    "item_name": null,
    "items_left": 0,
    "items_total": 3,
    "step": "Finished",
    "size_left": 0,
    "details": {
      "rpm": {
        "num_success": 0,
        "total_count": 3,
        "items_left": 0,
        "size_left": 0,
        "total_size_bytes": 6791,
        "num_error": 3
      }
    },
    "error_details": [
      {
        "traceback": "",
        "checksumtype": "sha256",
        "checksum": "435d92e6c09248b501b8d2ae786f92ccfad69fab8b1bc774e2b66ff6c0d83979",
        "error_type": "error",
        "fileName": "pulp-dot-2.0-test-0.1.2-1.fc11.x86_64.rpm",
        "downloadurl": "http://jmatthews.fedorapeople.org/repo_with_bad_read_perms//pulp-dot-2.0-test-0.1.2-1.fc11.x86_64.rpm",
        "item_type": "rpm",
        "savepath": "/var/lib/pulp//repos/repo_with_bad_read_perms//",
        "error": "HTTP status code of 403 received for http://jmatthews.fedorapeople.org/repo_with_bad_read_perms//pulp-dot-2.0-test-0.1.2-1.fc11.x86_64.rpm",
        "pkgpath": "/var/lib/pulp//packages/pulp-dot-2.0-test/0.1.2/1.fc11/x86_64/435",
        "size": 2359
      },
      {
        "traceback": "",
        "checksumtype": "sha256",
        "checksum": "6bce3f26e1fc0fc52ac996f39c0d0e14fc26fb8077081d5b4dbfb6431b08aa9f",
        "error_type": "error",
        "fileName": "pulp-test-package-0.3.1-1.fc11.x86_64.rpm",
        "downloadurl": "http://jmatthews.fedorapeople.org/repo_with_bad_read_perms//pulp-test-package-0.3.1-1.fc11.x86_64.rpm",
        "item_type": "rpm",
        "savepath": "/var/lib/pulp//repos/repo_with_bad_read_perms//",
        "error": "HTTP status code of 403 received for http://jmatthews.fedorapeople.org/repo_with_bad_read_perms//pulp-test-package-0.3.1-1.fc11.x86_64.rpm",
        "pkgpath": "/var/lib/pulp//packages/pulp-test-package/0.3.1/1.fc11/x86_64/6bc",
        "size": 2216
      },
      {
        "traceback": "",
        "checksumtype": "sha256",
        "checksum": "4dbde07b4a8eab57e42ed0c9203083f1d61e0b13935d1a569193ed8efc9ecfd7",
        "error_type": "error",
        "fileName": "pulp-test-package-0.2.1-1.fc11.x86_64.rpm",
        "downloadurl": "http://jmatthews.fedorapeople.org/repo_with_bad_read_perms//pulp-test-package-0.2.1-1.fc11.x86_64.rpm",
        "item_type": "rpm",
        "savepath": "/var/lib/pulp//repos/repo_with_bad_read_perms//",
        "error": "HTTP status code of 403 received for http://jmatthews.fedorapeople.org/repo_with_bad_read_perms//pulp-test-package-0.2.1-1.fc11.x86_64.rpm",
        "pkgpath": "/var/lib/pulp//packages/pulp-test-package/0.2.1/1.fc11/x86_64/4db",
        "size": 2216
      }
    ],
    "num_error": 3
  },
  "id": "2b36f5f0-7286-11e0-ae54-002564a85a58"
}
Comment 5 Preethi Thomas 2011-08-16 12:10:18 UTC 
Closing with Community Release 15

pulp-0.0.223-4.
Comment 6 Preethi Thomas 2011-08-16 12:22:23 UTC 
Closing with Community Release 15

pulp-0.0.223-4.