text:
Type: Improvement
Status: Closed
Priority: Critical
Resolution: Fixed
Affects Version/s: 1.8.4, 1.9.10, 2.0.1, 2.3.2, 2.4, 2.5
Fix Version/s: 2.6
Component/s: Administration
Environment:Redhat Enterprise 4AS running Apache 2.2.x, MySQL 5, and Moodle 1.8.x and Moodle 1.9.xbeta
Testing Instructions:
- Create a CSV to import data from
- Navigate to Home / Site administration / Courses / Upload courses
- TEST: Different combinations of settings
- TEST: Different combinations of CSV data (https://tracker.moodle.org/browse/MDL-13114?focusedCommentId=231131&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-231131)
- Make sure
- Expected errors occurred
- Update/Create/Delete/Rename, etc... worked when no errors occurred
Repeat using the command line tool.Create a CSV to import data from Navigate to Home / Site administration / Courses / Upload courses TEST : Different combinations of settings TEST : Different combinations of CSV data ( https://tracker.moodle.org/browse/MDL-13114?focusedCommentId=231131&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-231131 ) Make sure Expected errors occurred Update/Create/Delete/Rename, etc... worked when no errors occurred Repeat using the command line tool.
Affected Branches:MOODLE_18_STABLE, MOODLE_19_STABLE, MOODLE_20_STABLE, MOODLE_23_STABLE, MOODLE_24_STABLE, MOODLE_25_STABLE
Fixed Branches:MOODLE_26_STABLE
Pull from Repository:
Pull Master Branch:
Pull Master Diff URL:
Sprint:BACKEND Sprint 2
Story Points (Obsolete):100
Sprint:BACKEND Sprint 2
We're looking to Moodle for a replacement to Blackboard Enterprise. In order for Moodle to be a viable replacement we need a way to bulk create and delete courses based on CSV files. The bulk create courses is handled by use of a 3rd party module (uploadcourse.php created by Rory Allford). However once we import say 700+ courses there's no way to them bulk remove them at the end of a semester (removing 1 by 1 would be too time consuming). We would really prefer that this functionality be included within Moodle, as relying on 3rd party modules (no matter how good they are), inherits a risk if the module stops getting updated.
Gliffy Diagrams
Issue Links
- caused a regression
MDL-40898 windows unit tests failures in course upload tool
- Closed
- has a non-specific relationship to
MDL-40737 Prevent any alteration of frontpage from tool_uploadcourse
- Closed
MDL-40728 Improve UI documentation for tool_uploadcourse
- Closed
MDL-38501 Create a "Bulk course action" menu, allowing several operations (backup, delete, reset, change settings...)
- Open
- has been marked as being related by
MDL-40903 Split the persistent cache option into logical parts.
- Closed
CONTRIB-605 Bulk Delete Course Utility
- Development in progress
- is duplicated by
MDL-14341 Admin: Upload Courses and Course Categories from CSV file
- Closed
MDL-24881 Bulk Create Courses in Moodle 2.0
- Closed
- will be (partly) resolved by
MDL-17017 Admin: Evaluate file structure for administrative functions for consistency
- Closed
As you can see from the above nearly three-year discussion, there is tremendous interest in bulk course creation. I was pursuaded of its importance by a presentation by A.T. Wyatt at the Oklahoma City Moodle Moot in June 2008. At Wyatt's university, a course is created for each teacher training student to serve as a preservice teaching porfolio. FYI Wyatt uses a script written by Mark Pearson and posted in http://moodle.org/mod/forum/discuss.php?d=30126. This script is no longer being maintained.
The Modules and Plugins entry http://moodle.org/mod/data/view.php?d=13&rid=395 ("Bulk Course and Category Creation Tool (Upload CSV)") which seems to be most popular is also no longer actively maintained by its author, Rory Alford, and zip files have once again begun to appear in the forum. Rory's tool can create categories (and subcategories) as well as courses.
Is there any way that this feature can be made standard? For example, Site administration -> Courses -> Upload courses.
I have to change the attribution to Jeff Church.
Mark Pearson and I did work on this, and Mark has some good documentation on it. But Mark did not write the script.
But I think this is sorely needed in the standard instance. There are a lot of people who have the need to do bulk course creation (and deletions). Mike Churchward had some good ideas on this also; I visited with him about it last September.
I will 'third' this as an important issue to address and would support this being bumped from minor to major priority. Besides it's inherent value, having a bulk course upload script and procedure formalized and incorporated into the Moodle core would also be a first step towards formalizing integration with institutional ERP systems. I would like to see some discussion about whether the data format should be simplified and some definition of what the expected behaviour of the script should be (eg not duplicating courses already present). It seems to me that this process should not be an onerous task since the script is already there and working (though I have not tested it with 1.9).
The bulk course creation tool still works in 1.9. I was not aware that the script is no longer maintained. This is bad news for us as we require it's use. I really hope that this makes it into the default Moodle install. Why not? All the work is basicaly done, it just now needs to be maintained for future Moodle releases. This has been a major frustration for us in the Moodle project. Fairly complex features like Moodle Networking are developed, but simple things like batch course create/delete are not. This seems like such a basic feature....
I have tried to review the discussion in the forum from time to time on the bulk course add and delete admin scripts as there has been a good deal of confusion from time to time as to what's what.
As I understand the situation now, there is an upload utility in M&P that came via Jeff in M&P and a delete utility that was also posted by Jeff and then massaged by Chad posted to the forum, but not added to M&P, and both add and delete utilities now work in 1.8 and 1.9. As I understand it, these utilities have been functioning appropriately and I don;t think I have seen anything in the way of problems lately.
I created a tracker issue to add the delete utility as a new M&P page or as part of an add/delete utility M&P page pending addition of the utility to Moodle core.
The discussion still seems to get convoluted and I would like to see perhaps the following steps taken:
1) clearly document the source of the bulk add and delete course utilities that were offered by Jeff and then tweaked for 1.9
2) clearly identify both of the bulk course utilities in M&P, preferably packaging them together and offering them from the same M&P page
3) review same to determine what needs to be done to incorporate these utilities into core
4) incoporate them into core.
|Original Value
|New Value
|This issue has been marked as being related by CONTRIB-605 [ CONTRIB-605 ]
Marc - OK, I think I am ready to start making some progress on this. I think this is where we stand.
1) The course for the bulk course upload done by Jeff is available at http://moodle.org/file.php/5/moddata/forum/33/427997/bulk_course_tools.zip and the M&P entry which should have its download link updated is http://moodle.org/mod/data/view.php?d=13&rid=1536&filter=1. As for backup and delete, Moodlerooms has a course management block that has these functions available at http://development.moodlerooms.com/file.php/1/Moodlerooms_Releases/Course_Management_Block_for_1.9/coursemanagementblock19.zip and one provided by Robert Russo in CONTRIB-605 is available at http://download.moodle.org/download.php/plugins/blocks/backadel.zip. I've asked for Justin Hardin, Robert Russo and Ashley Gooding to help me understand the relationship (if any) between these two similar blocks. In speaking with Dan Marsden, I think the goal we are looking for is to have it as consistent as possible with other areas of import and functionality in Moodle so it would be integrated under Administration rather than a block.
2) Once we have identified the preferred backup and delete tool we can package them together into a single patch.
3) Once we have a patch we can then move forward on incorporating these tools into core.
Peace - Anthony
Should we change the affects version to 1.9? I do not anticipate doing this for 1.8 but perhaps we can backport it once we have it working.
Hi, yesterday (26-4-2009) we have just release a free tool for realizing bulk actions on moodle. This tool allows you realize bacth process for deleting courses, creating courses, assaign teachers to courses and deleting users.
We develop this applicarion as a result of our work Universidad Adolfo Ibañez. They needed a tool the perform the change of semester (delete oldo courses, create new courses, assaign teacher to the new courses and delete old users) and this application is the result. We tested duringa the past year deleting more than 700 courses each semester and creating over 800 courses each semester without problems.
This is our first release of this applicarion and we hope to continue improving it in time. Please take a look at it at www.moodleadminpro.com
Sebastián Espinosa
MoodleAdmin Team
Here is a minor fix of uploadcourse.php that restores the ability to add multiple teachers to a course
It seems that this has been dormant for a short time. I would really like to see the Bulk Course Upload tool for 1.9 (one version with all of the features of Rory Alford & Jeff Church's contributions) added to and maintained in the Modules and Plugins page. I am not sure if this capability is being added to 2.0 or not but nevertheless it needs to be maintained for users of 1.9 who are not going to make the jump to 2.0 for some time.
I have incorporated the following fixes into the tool and am willing to volunteer to maintain it in the Modules and Plugins area for 1.9 as best I can until it is added to the 1.9 core. I am not a true programmer but am capable of making adjustments as people on the forums contribute as well as at making small additions myself and am willing to do the legwork to keep this updated for 1.9.
Updates to the tool I originally obtained from Jeff Church here (http://moodle.org/mod/forum/discuss.php?d=91850)
1. Small changes to the text strings to make the explanation of the fields easier to understand and/or to update information as field possibilities have changed.
2. Added the ability to use the 'enrol' field in the CSV file to set the enrollment setting for courses.
3. Made adjustments explained here (http://moodle.org/mod/forum/discuss.php?d=91850#p572025) to enable the tool to enroll multiple teachers.
And will also update the code to remove the bug explained by Jeff Sherk here (http://moodle.org/mod/forum/discuss.php?d=91850#p405748) as well as some of the field 'expansion' fixes explained by Fernando here (http://moodle.org/mod/forum/discuss.php?d=27333#p405593).
Is there any way that we can move forward a little on this and get one entry in the Modules and Plugins area that points to a file instead of to a forum? Again, I'm willing to help out by maintaining this in the M & P area.
This version includes the fix to allow for multiple teacher enrollment and also the addition of the enrol field so that the course enrolment plugin can be set via the tool.
|uploadcourse.php [ 20298 ]
*Bump... Any news on if this functionality will be added/avaialble for Moodle 2.0? Currently evaluating Moodle 2.0 RC1 and sadly there appears no way to bulk create courses.
Sorry, I'm not sure. I haven't experimented with 2.0 enough yet to even know that the capability doesn't exist. I'm a bit surprised that it wasn't added though....!
I'll try and work with it a little and see how far off this tool is in terms of being usable for 2.0. If it requires serious modification I'm afraid it will be beyond me but I'll see what I can do.
Anyone else know what the plan is for bulk creation of courses in 2.0 onward?
|jira [ 24496 ]
|MDL Workflow [ 42951 ]
I'm adding 1.9 and 2.0 affected branches so that this does not fall through the cracks. I'm also adding the Fix Version of DEV backlog to hopefully get work on this moving forward. Peace - Anthony
|Fix Version/s
|DEV backlog [ 10464 ]
|Affects Version/s
|2.0.1 [ 10420 ]
|Affects Version/s
|1.9.10 [ 10407 ]
This is going to be a key component in getting my organization from switching from BB to Moodle. I know that there was a patch in 1.9 backwards but there is nothing yet for 2.0 (which is what they would want to switch to). Unless someone can fix the old file to work in 2.0.
|MDL Workflow [ 42951 ]
|MDL Full Workflow [ 71356 ]
Anthony: any progress?
I notice there is some code in this thread: http://moodle.org/mod/forum/discuss.php?d=91850&parent=744931
I had a quick look at the script in the forums and unfortunately there's no way it can make core in the current condition. It's not even remotely following our coding guidelines or existing Moodle code. Just some quick examples:
direct access to MySQL at the top (not using our DB layer).
doesn't use core API to do the work
no string translations
no phpdocs
Is there some kind of spec on exactly what this function should do? Can somebody start one?
Just a quick note to mention this would be very useful for myself as well. If I can help in any way that would be great!
Elevating the priority on this so it is more likely to get some love post-2.1.
|Minor [ 4 ]
|Critical [ 2 ]
|Martin Dougiamas [ dougiamas ]
|David Mudrak [ mudrd8mz ]
As someone who's moved over from being a Blackboard admin I'm missing this functionality badly.
As Martin mentions above, some spec of the functionality would be nice to have before we start work on this. Let me suggest to follow "do one thing and do it well" approach instead of trying to combine course creation with enrolments etc via one CVS.
We seem to be stalled here on something that
1) a lot of users agree is an essential item of functionality
2) we had a running version of for 1.9
3) is fundamental to upscaling Moodle to large organisations
Martin's question was "Is there some kind of spec on exactly what this function should do? Can somebody start one?". Clearly the currently available fixes are not meeting the tech spec that is required for 'growed up' Moodle - however these can act as functional models of what is required. Hence the 'what it should do' is addressed by existing fixes and patches.
For my money I simply need
- to create (or draw down from our local registry) a list of courses
- to manipulate the list to add/edit shortname, course code etc.
- to be able to set core settings for courses (format, number of topics, start and end date etc.)
- to be able to add at least one editing teacher via the list
- to upload to the moodle site and cross check for duplicates
Can we flesh this out towards a fuller spec ?
Here's an example of a 'quick'n dirty .csv upload that I was using for 1.9 using one of the patches
2011.1 PG Courses,manual,0,0,topics,011.1 719 Cost Price and Revenue Management (PG),0,0,0,0,0,0,5,locked,011.1 719 Cost ,0,0
In my view the current urgency is to establish the bulk course creation, for those about to join Moodle, and for those of us trying to start a new teaching year on 2.0/2.1 - bulk course removal isn't going to be an issue for some time.....
Do you need a CSV file?
We're doing this for years in XML with an IMS Enterprise file.
Maybe this can help to someone: http://www.stylusstudio.com/csv_to_xml.html or http://csv2xml.sourceforge.net/
you can convert the CSV file to IMS file
If this gets implemented could we please first create real CSV import/export library that supports different separators and standard quoting?
"If this gets implemented could we please first create real CSV import/export library that supports different separators and standard quoting?" Petr, please translate.
"Current implementation for CVS upload to create accounts is inadequate"
"Old methods need to be fixed to new methods"
"If we fix some things now it will improve things and make other CSV importe easy"
"We now use only one separator and non standard quotes"
"This means a further delay since 2008 when this was first proposed"
"What is currently used is not a real import/export/library" ????????????????
I'm not a PHP coder. But it just seems that this should be not a big deal. (is it?) What saddens me is that people are creating tons of hacks (which uses time) but no-one is putting in the effort to co-ordinate and focus efforst to get this into standard Moodle.
What is involved in doing what you ask Petr? A day of coding? Does MoodleHQ want it or see the need for it?
</ramble> </rant>
I've written a small proof-of-concept patch for 2.1 which borrows very liberally from the bulk user upload page. This patch is deliberately incomplete. It's missing most forms of error-checking and validation. It doesn't display results unless something blows up. It does, however, use a standard interface and standard internal functions, which I believe addresses a number of the basic concerns raised by Martin and others.
Is it worth taking this patch further? If not, I don't mind the wasted morning.
Patch here on github: https://github.com/mackensen/moodle/compare/mdl-13114.
As far as I have followed this question, I realize that course creation and user enrolment depend on very singular situations in the different schools and universities. That is why I started to program my own import script with a GUI AND an unattended mode version. See here: http://moodle.org/mod/forum/discuss.php?d=128317
With Anthnoy Borrow I considered to modify my script so as to:
- import/create categories (adding the function into the GUI, like upload users)
- import/create courses (adding the function into teh GUI, like upload users)
- import/create/enrol users (as already in the GUI)
- delete courses (adding the function into the GUI, like upload users)
- all-in-one script/function to do all this with one and the same .csv File and script (adding this function to the GUI, like upload users)
- all-in-one script that can be executed in unattended mode, during night, through a cron-job (already working for me and my needs. Our Workflow:
- courses maintained in our school administration system are exported nightly to a windows-share in .csv UTF-8 format. The csv file contains all the columns you can see in my template in the above thread, short everything needed to create new courses and enroll the given users in their respective roles
- the Moodle cron-job executes my uploadusersandcourses.php script every night writing back to the share a text file with error-codes like "course xyz already exists", or "category xyz does not exist", or "course xyz created", etc. so that we know exactly what the script did during night)
My script only lacks deletion of courses as at our university we have to consider also archiving of old courses. So we could define what archiving means: copying only the backup-zip-File to a certain backup-server oder backup-archive-directory on the same server, and so on. Archiving could also mean MOVING the whole course to an archive category in the same Moodle server. Actually I move the courses through the GUI to an archive or toDelete category where I leave them hidden and without teachers for another 6 months before deleting them definitively or letting them in the archive category forever.
But my script has also some more features (realized and planned):
- course copy/creation from a given template (backup-zip-file) of an existing course (working fine)
- rename course from old name to new name (because we have courses that change only their name before starting over for a next semester or academic year, not programmed yet, but feasible, scheduled for next release)
- archive course (has to be defined, not programmed yet)
So we just had to join our efforts to get existing code of different programmers into the GUI and core of Moodle. I hope to find enough time before winter to realize the above listed functions separatedly and in an all-in-one script.
And this gui already programmed a course upload script for Moodle 2.x: http://moodle.org/mod/forum/discuss.php?d=173705
Hi Rosario,
as I wrote at some comments above, we've been doing all this kind of stuff since 2005 with XML (not CSV) with an "IMS Enterprise" file that is built-in in moodle (you don't need to maintain it) and it's an standard (we didn't want to reinvent the wheel and yes, we know some other people feels more comfortable with csv).
The "IMS Enterprise" lacks the course creation from a given template (but you can rename a course, move it from one category to an other...), so I'll study the way to do this with the "IMS Enterprise" file, this will save lot of work to our staff.
Eduardo, I think the main reason behind csv is that we usually take the courses list from another system. Some of these may be made with old languages and can be a PITA to export data to xml. Ie: Fox and Clipper. It's easy to get a .dbf and convert it to csv and when the original programmer or source code from that system is not available it's a problem.
To transform the csv to xml is a one-step procedure in excell using the concat() function. This is what I do when some teachers want me to create 50 courses, they give me a csv file and I add the last column (this is a short example and nobody needs a programmer to do this, but if you have one, you can automate this step in a couple of hours):
|Content of file.xml
|Maths for dummies
|Physics for dummies
and so on.
What I want to said is that lot of people have been waiting for an utility that is already made. I know that the people is busy and maybe they don't have time to read the IMS Enterprise specification, what I'm telling is that the solution to their needs has been there for years and that someone, like me, can tell them where it is.
Alejandro, Eduardo, I DID consider IMS Enterprise Import before I decided to program my own uploadusersandcourses.php and I made some posts at that time in the moodle forums. The problem is exactly as Alejandro says, other systems used for course and user management are all able to export into simple text or csv files, but not to IMS Enterprise format. So even if we/I could automate a csv into IMS transformation, this would be an extra step to control and maintain over years. And my intention was to offer the same GUI interface Moodle offers in upload users which uses csv files and to separately combine all the code into one all-in-one script you can schedule during night.
Eduardo, I found your guidance on using the IMS Enterprise import to be very helpful, so I'd also like to add my thanks.
However I think we still need a csv upload/delete.
I'm just starting Moodle 2.x , and trying to visualise how it might be scaled to Enterprise level. I seem to be stuck at these issues:
IMS Enterprise does not delete courses
IMS Enterprise requires a unique course before creating a category (At the start I only want an empty structure)
IMS Enterprise does not create sub-categories
IMS Enterprise does not support course properties (number of sections etc.)
It seems to be a fundamental restriction that IMS only knows about generic IMS things, whereas a CSV import could be targetted at Moodle concepts.
Thanx to you two too,
I missed some functionality to IMS Enterprise too (groups and grouping for instance) but IMS can be extended in any way you need. We'd extended the use to support WebDAV and SSO from our intranet.
I'll post a suggestion to the developers to do the same job to IMS and CSV import. This makes no sense to have different functionality between two import systems from a file (csv, ims, ods...)
Hi -
This may or may not be of any use to people on this thread, but I create a basic course uploader for 2.x modelled on the user uploader, as I needed something for NZ schools - it's available here https://gitorious.org/moodle-tool_uploadcourse/moodle-tool_uploadcourse for the curious.
Piers Harding.
Piers, I'm trying to use your course upload tool with Moodle 2.1.2, but I am unable to get Moodle to recognise it. Does it work with 2.1.x? I put the files in the relevant place, but nothing I do kicks off an installation so it never shows up on the menu.
Sean K
Hi Sean - sorry, but I have only seen this just now. It is an admin tool plugin and these are only picked up from 2.2 onwards. I've recently updated the plugin to use templating from other courses, or a Moodle backup.
Piers Harding.
Absolutely unbelievable that a tool used by universities across the world does not come with state-of-the-art mass import / export / handling tools... hopefully this is just a misunderstanding or I have just a bad dream.
Hi John,
You get used to it after a while - Moodle still seems to be dogged by 'small system' perspectives.
However, if you've moved to 2.02 plus you can try Piers Hardings contribution - this got us out of trouble last semester, and I believe that Piers has improved his plugin further have a look her: https://github.com/piersharding/moodle-tool_uploadcourse
John - I wouldn't be too harsh about it. Two things have happened in 2.x that make it possible to build better tools for managing courses -
- the back up/restore engine has been improved immensely (I believe, thanks to Eloy)
- there is now a great admin/tools plugin architecture
These two things have made it possible for the creation of https://github.com/piersharding/moodle-tool_uploadcourse , and it is probably only a matter of time before something like this is available in core.
Have a go, and let me know how you get on.
Piers Harding.
Excellent tool Piers, thanks so much for developing this! This is going to help us immensely. Quick question - do you know of a way to bulk create course categories as well? If not, no biggie, much easier to just create the categories first and then generate the courses. Thanks again!!
Hi Susan -
No I don't know of a facility for bulk creating course categories - something to think about developing as a complementary tool.
Piers Harding.
thx. just trying the delete function and it doesn't seem to be working. I am using deleted with a value of 1
any ideas?
.. oh never mind I figured it out!!
Does adding the teacher or editingteacher field allow you to actually assign that role to someone via the csv file?
- I see you can do that using upload users. I should probably stop commenting here ... sorry all - last one
Hi -
I just built a course category uploader here - https://github.com/piersharding/moodle-tool_uploadcoursecategory
Piers Harding.
Hi Piers,
I am using your tool to upload mass amt of courses but every once in a while it hiccups right after I upload the file and gives me an error:
[Error occur during loading CSV file! More information about this error]
which leads me here: http://docs.moodle.org/22/en/error/moodle/csvloaderror
I can't make heads or tails of it. It seems if I do smaller chunks it works ok.. but I can't find a common denominator here. The original data it hiccupped on uploaded fine in smaller chunks.
Any thoughts on why it would be tripping up?
Hi Susan - It might be an idea to take this off this tracker as what I've developed isn't an official Moodle effort (of course, everyone is welcome to it), so you can contact me direct - piers at ompka dot net. Do you think it could be a timeout (or memory) related problem? How long does it run for before it dies? If you take the ones that are missing and continue processing them, do they now process ok?
Piers Harding.
It's quite useful to have this here so that others who are interested can benefit. We can always opt out if the issue becomes too busy.
I've been tripped up by content length before.
If content is too long for the receiving field .csv uploads sometimes seem to crash, but with no errror message.
looking forwards to reports on the status of the program
best Paul
Thanks for your feedback! Would it make sense to move this discussion to the forum?
How difficult would it be to add enrollment options to this? For example, I have created 500 new courses and it would be beneficial if I could pre-load each course with the self-enrollment option and an initial password. (I'm not asking for much am I )
Relatively few users deal directly with the tracker so I strongly recommend taking prolonged, non-programatic discussions to the forums.
Hi Susan - support for creating enrolment options (not enrolling users) is something that I would like to add, but it is a reasonably complex piece of functionality, so it is not likely to be done for at least a few weeks. It would need to be flexible, as courses can have multiple enrolment methods, with each enrolment method having configurable options.
Piers Harding.
Hi -
I've pushed a set of changes to https://github.com/piersharding/moodle-tool_uploadcourse to handle population of enrolment methods - please see the README for details.
Piers Harding.
Has anyone modified this code so that it can be invoked from the command line, say from a cron job, without having to use the Moodle interface? If not, can anyone more knowledgeable than me comment on how much work that might be? Thanks for a great piece of code!
Hi Patrick - It hasn't been done, and it is probably a day or twos work to do. The majority of the logic could be pulled out and remain the same, but the front-end code that deals with input parameters for defaults, switches, and file source would have to be hauled out and remodelled.
The other major piece of work would be coming up with a sane output format for all the information that gets reported back to the user on job completion.
I wrote a framework for this kind of thing for Moodle 1.9 called moodlectl, so some of this could be pulled out, but it's still more work than I can do in the short term - I'll put it on the to do list. The very next thing that will be asked for is the same for User upload .
Piers Harding.
Just about to work on something like 47 new courses, and in a slightly grumpy mood throught I would check in to see if this 130 vote issue dating back over four years and lots of posts and hacks over the years had had any traction for being integrated. Not today. I wonder if it is quicker to create a bunch of these courses in a local install and then restore them one by one in the real destination?
Just pre-testing this ahead of uploading. Piers - am I right in saying that the create course via existing course template does NOT automatically pick up the number of sections from the old course?
many thanks
Hi Paul - that is what I would expect (NOT auto pickup) - are you experiencing something different? Can you give me an example?
Piers Harding.
Hi Piers,
This may get a bit detailed - could we switch to http://moodle.org/mod/forum/discuss.php?d=173705&parent=854771 ?
Is there any movement on adding Piers Harding's code to core? There are a few things that I've noticed that need to be fixed to make it even possible to add to the core.
For some reason I decided to build basically the same thing as Piers but very basic. Its built off the same setup as the Upload Users. It gives all the options that you get when creating courses manually.
Repos is here on github https://github.com/henesnarfel/Moodle-Upload-Courses
I feel that the work Piers has done was very good but just needs some updates, modifications, and cleanup to maybe be added to core someday. I think that to at least start getting the functionality into the core we should keep it simple. Then add to it as the community asks. That's why I created my very basic version.
Let me know your thoughts
The problem with this tracker issue is that, although the initial idea is still valid, it wont be going anywhere at all as long as the Moodle versions, listed at the top of this page, stay as they are. You may not be aware of the fact that those version are no longer supported other than the odd security fix for Moodle 2.0.
I've just added you as a watcher, as I feel
MDL-13114 will slip off the radar if not updated and triaged.
|Affects Version/s
|2.3.2 [ 12353 ]
|Affects Version/s
|2.3.3 [ 12373 ]
|Affects Version/s
|2.4 [ 12255 ]
Thanks to the people who have contributed to this issue. It looks to be a popular improvement worth considering.
|Issue Type
|New Feature [ 2 ]
|Improvement [ 4 ]
|patch triaged
|Affects Version/s
|2.4 [ 12255 ]
|Affects Version/s
|2.3.3 [ 12373 ]
@Michael, do you think anything will happen with this? Has anyone had a look at the code James has worked on?
Just curious, before I go ahead and manually create 92 courses.
I've checked out https://moodle.org/mod/forum/discuss.php?d=173705 but the latest activity is in August.
@Derek, Just FYI I am using the latest code from Piers Harding referenced above to bulk upload courses to Moodle 2.2. I am also using uploadusersandcourses_silently.php by Rosario Carcò referenced above for Moodle 1.9.
@Patrick. Thanks. The reason I was re-visiting this issue is that I had some work to do in Moodle 2.3.2. Do you know if there is anything for this version yet?
@Derek, I have not used it but Rosario did post a version of his code for 2.3 in this thread:
He calls it a beta version.
I can say that no one at HQ is working on this currently and it won't be in 2.4.
With the popularity of the issue, the fact that it has a patch, and the fact that it doesn't replace existing functionality, I think it is a good candidate for the roadmap. I've recommended it.
Hi Michael -
Which way is this leaning - is it towards the CSV upload admin tool that I've written or something else? If it is a matter of working through an acceptance process then I'm happy to work with a core dev on it.
Piers Harding.
|David Mudrak [ mudrd8mz ]
|Dan Marsden [ danmarsden ]
talked with David/Michael and have re-assigned this to myself - I'll pick this up after 2.4 has been released/branched and will see if we can get this into core.
Dear all, my offer is still valid. I reengineered my uploadusersandcourses.php and uploaduserandcourses_silently.php code to be used on Moodle 2.3
It is working in production for us since August/September.
As adding third party code/modules has become even easier in Moodle 2, there is no need to ask to incorporate such code into core, unless HQ wants to do it. We can have those pieces of code neatly added as I already did with my uploadusersandcourses.php - See my thread here, download and test it at your will. The GUI-version works almost out of the box, but read the according readme-file first:
Thanks Rosario - I think there's general consensus that it would be good to include a tool that performs this functionality into Moodle core but this will require a final +1 from Martin D before anything lands in core.
I'll review your code and Piers code after 2.4 is branched and will check with Martin (the number of votes on this issue make this an easy case to make). At this stage I think adding this as an admin tool is the best approach but I haven't reviewed any of the code yet. (will probably do this sometime in Jan/Feb)
Dan, thanks for the great news. As you plan for Jan/Feb, this could also be my own pace, as I still have to reengineer also myCourses and siteNavigation blocks of mine to be used on Moodle 2.3 first. As I decided to use an all-in-one strategy mainly to use the _silently.php version as batch/cron job, we would have to decide whether to leave/put the logic of WHAT HAPPENS WITH THE COURSE either in the csv-file, as I actually implemented it with my ACTION-Column, where you write NEW (to create a new course), COPY (to copy an existing course), RENAME (to rename a course for the next semester/term, not implemented yet), ARCHIVE (to archive an existing course, but not defined yet, and not implemented yet), DELETE (to delete a course, not implemented yet) and so on. OR whether we wanted to extract/move this logic out of the csv-file into some Web-Form-Options in the GUI-Version, which means we would have to implement something like:
- ARCHIVE EXISTING COURSES (whatever and however this will be defined, eg. move only the mbk-zip-file into an ARCHIVE-Course or Category)
- DELETE EXISTING COURSES (which could also be defined as archive first, then delete or move from the category it is in)
- and so on
uploaduser.php itself is a hybrid thing, containing columns with ACTIONS and offering a lot of GUI-options at the same time. So I think that deciding what way to go (complex csv file and lean GUI or simple/lean csv file and complex GUI options) is even more difficult than to reengineer our (the above mentioned) codes. If we want to stick to the idea of ONE csv file for everything, so that it can be used also in unattended batch/cron mode, AND in the GUI, we would have to proceed on my way I think, i.e. complex csv-file an lean GUI.
Hi -
Just split the code out so that there is a command line version as well as the admin/tool - thanks to Moshe Golden for help - https://github.com/piersharding/moodle-tool_uploadcourse
Piers Harding.
|Fix Version/s
|DEV Current [ 12256 ]
Looks good (in principle) to me - needs a peer review from DEV for 2.5!
|Open [ 1 ]
|Waiting for peer review [ 10012 ]
great - thanks Martin, updating this with Piers latest version - I'm pretty busy over Feb/March so it would be good if someone from HQ has time to peer review - if not I'll try to squeeze something in.
|Pull Master Diff URL
|Pull Master Branch
Just pushed a commit to pass local/codechecker. Will do the same for local/moodlecheck, and comment back here.
Does this ability include updating the course start date and corresponding dates in activities? Looking forward to seeing what this tool has to offer!
At the moment, you can process updates for the course startdate. It is a little uncertain what this whole feature will end up looking like, as it still needs to be reviewed/evaluated for acceptance.
Piers Harding.
Just pushed another commit to ensure that it passes the local/moodlecheck as well to https://github.com/piersharding/moodle-tool_uploadcourse
I just skimmed the code (I'm afraid I don't have time to do a proper peer review now, but thought it better I comment about these things I spotted in case nobody else can get to it quickly):
- The cc_ prefix for the various functions would need to change to something a bit more verbsoe for core inclusion. I'd advocate moving to something OO like tool_uploadcourse and static methods on that. But i've not looked deeply.
- settings.php uses moodle/site:uploadusers, thats not right - its probably best to introduce a tool sepcific capability
- cc_process_course_upload should be split up - its 1000 lines long!
- Be good to see some unit tests
Hi -
1 and 2 are easy to fix.
3 presents a bit more of a challenge as uploadcourse is modeled on uploaduser - cc_process_course_upload() is just the equivalent of index.php for uploaduser - wrapped so that it can be called either interactively or via the command line.
4 unit tests can be built with no difficulty when the decision is made whether to pursue incorporating uploadcourse. It will be easy to pump test data into cc_process_course_upload() for this purpose.
Piers Harding.
I tried to upload the example from http://docs.moodle.org/24/en/Bulk_course_upload#Example_2st_record_in_csv_file
This is what I tried to upload:
Greatest Course;GC101;Education Portfolios;1;;University Portfolio;topics;0;0;Owner;Owners;Visitor;Visitors;1/14/2008;10;15728640;0;0;1/12/2008;1/12/2008;portfolio;0;0;0;;;;0;2;1;1/14/2008;5/10/2008;0;0;10;editingteacher;firstname.lastname@example.com
This is the error I got:
"sortorder" is geen geldige veldnaam [translation: sortorder is not a valid fieldname]
Are there translation issues? Can you supply an example .csv file for me to test?
Hi Richard - the example you refer to is for a different course uploader, so I would not expect it to work. You can start with the documentation here https://github.com/piersharding/moodle-tool_uploadcourse and a simple file example is:
Piers Harding.
Thanks Piers, that explains it. Hope you can get this in the core 2.5 as there are a couple of versions going around now, hard to keep track
I am very busy these times. Can you quickly tell me whether the actual code will also create missing categories like "english-courses/beginners/first-semester" and the like, i.e. the whole category path where to put in the course we are creating, if that category-path does not exist yet. Marco Cassisa has implemented that code in my uploadusersandcourses.php and I will review his code asap. In a certain sense I am still advocating for my split up functionality at GUI level, as I proposed higher up already, sort of
ARCHIVE EXISTING COURSES (whatever and however this will be defined, eg. move only the mbk-zip-file into an ARCHIVE-Course or Category)
DELETE EXISTING COURSES (which could also be defined as archive first, then delete or move from the category it is in)
CREATE NEW CATEGORIES (whole Path if it does not exist)
COPY EXISTING CATEGORIES (including every subcategory/whole category path, including every course in cats and subcats
and so on
While still integrating everything into one silently uploadusersandcourses which becomes bulk_course_actions for unattended command-line/cron-jobs. Apologies if this is already the idea but I did not have time to read the docs on Pears' link.
Talking about bulk course actions, i've created MDL-38501.
Piers : I've tested your tools to automatically create courses (or course categories), and they are useful !
To be more usable, i think that some things should be changed : in README, you show which fields can be specified, but it would be really useful to have the list of required and optional fields (like in Upload user documentation : http://docs.moodle.org/en/Upload_users ).
It seems that actually several fields are required by your tools (like idnumber), but are not (required) in Moodle. I'm not sure it's really a good idea. Would better let only Moodle required fields to be required.
|Fix Version/s
|Fix Version/s
|DEV Current [ 12256 ]
Hello. This issue is missing testing instructions. Please add a description of how it should be tested remembering to include, where appropriate, verification that no existing functionality was broken.
|Affects Version/s
|2.5 [ 12452 ]
|Affects Version/s
|2.4 [ 12255 ]
|Story Points
|Ranked higher
|Original Estimate
|0 minutes [ 0 ]
|Remaining Estimate
|0 minutes [ 0 ]
|Waiting for peer review [ 10012 ]
|Peer review in progress [ 10013 ]
|Peer reviewer
Hi everyone,
I spend some time reading through the code and trying out the plugin, it's a great job, thanks everyone!
My main concerns about the code lead more towards the maintainability and robustness of the code. Currently it is really procedural and there are bits and pieces of code duplicated in different areas (lib, index.php, CLI), they could certainly be merged into one single place. Also, the CLI and the UI differ a bit in their implementation, and would be hard to maintain as both side should benefit any further improvement.
I would highly recommend to wrap all the functionality in a class, which would include its own constants, different methods for each different operations (detect enrolment settings, restore a course, reset, ...), keep track of the amount of courses updated/deleted/created, the upload mode, etc.... It would also be easier to perform the checks like "If this course exists, then update the shortname and consider that it does not exist", and easier to perform the capability checks for specific fields (moodle/course:changecategory, moodle/course:changeidnumber, moodle/course:changeshortname) if need be.
But most importantly, there is currently no Unit Test coverage, and while it would surely be possible to write some basic tests with the existing code, having a class would make it a lot easier and granular.
In regard with the current functionalities, I wonder if we should really allow for course deletion as part of uploadcourses tool. Perhaps deletion should be a sub part of the plugin and completely distinct from the upload part. But then, maybe the module should be renamed 'Bulk course operations' or something... food for thoughts.
About the current patch
Here are some comments based on a use in the latest 2.6-dev.
Need unit testing
We do need a serious unit testing to prevent horrible things happening on existing courses!
Use of deprecated objects
- make_categories_list() is deprecated.
- get_context_instance() is deprecated.
- PARAM_MULTILANG is deprecated.
Capability checks
I am not sure about the current policy in regard with course restore and the capabilities associated to it, but should there be more granular checks when updating a current course? Some examples are:
- moodle/course:visibility
- moodle/course:reset
- moodle/course:changesummary
- moodle/course:changeshortname
- ...
Also, at the beginning of the script, the user requires course:create, course:update and course:delete but that might be too restrictive if the user is not trying to delete or update courses.
More importantly, the tool needs to define its own capabilities and stop using 'moodle/site:uploadusers'. Also in settings.php, we should definitely not make use of 'moodle/restore:restorecourse', but of the soon-to-be-introduced capability.
Needs clarification
- What is $bulk? It does not seem to be ever set to something else than 0.
- standardshortnames is quite ambiguous. What does it do? Shouldn't we ALWAYS clean the shortname?
- When is $backupfile not empty? And what is $roles in the logic associated to it?
- Is $CFG->keeptempdirectoriesonbackup relevant here, as far as I'm concerned, we are not really backuping a course, we're using this method to restore some content.
- The case where $optype === CC_COURSE_ADDNEW then if ($existingcourse) should never happen. Around line 500.
- In form::get_data() what is this description field?
To consider
Here are some comments on things that I think we should consider. Also, as this script could eventually process a lot of courses, keeping in mind performances is important.
- I am not a fan of oldshortname, could we consider keeping the existing shortname in its column and having another column rename_shortname_to or new_shortname?
- templatename considers none as empty, this should be avoided and replaced with a proper empty value.
- It seems quite expensive to extract a backup just to make sure it's valid. Shouldn't we extract it once and for all as it's going to be used by each course?
- I dislike enrolment_1 then using enroloption_1, I think it's more clear to use enrolment_[0-9]<option>. That would also allow for more same settings of the sort _anotherthing_1_option later on.
- Role renaming should be applied to all existing roles, not only the default ones.
- Deleting the course could happen right after checking if the course exists, rather than later.
- Support multilang filter. Ie: <span lang="en" class="multilang">English</span><span lang="fr" class="multilang">French</span> Cat. Detecting the course category does not support it.
- Speaking of categories... even though category IDs are harder to work with, they would solve any potential issue in resolving the path. Also we could have an option to fallback on the default one if the ID is not found. (Or category ID number, or one or the other)
- In the same was than 'Allow deletes' or 'Allow renames', there should be an 'Allow resets'.
- We should clearly define a list of mandatory fields for each upload type.
- There is a die() in the middle of the process when a backup could not be restored for some reason. I don't think we should halt the whole process, but perhaps just skip the current course.
- Do we really want to restore from another course, and then restore from a course file, and then reset eventually? Those 3 actions can be executed in a row.
- When restoring from a file, do we ensure that we do not restore user data? If not, should we?
Minor comments
- We usually only have one @copyright per file, the person who created the file writes his name there and it does not change: http://docs.moodle.org/dev/Coding_style#.40copyright
- Could use _DIR_ with ../ instead of dirname().
- We do not use underscores in variable names.
- in if (in_array($field, $stdfields) or in_array($lcfield, $stdfields)) the later is enough, our fields should never use caps.
- Adding support for Summary files (new in 2.5)
- Form headers addElement('header') should have a name in order to be collapsible.
- ccshortname help button mentions to view the 'help' for examples, what help?
- There should not be any default 'grouping' as they are defined in the course itself.
- Do we need to flush some cache after/during the process?
- The dropdown to select a course to use as a template could potentially contain hundreds of courses, it would make it unusable. Perhaps, we should not support this and rely on the uploaded restore file only.
- The default course settings should reflect the new tree/structure/sections/labels/values introduced in Moodle 2.5
- The use of file as a reference to the CSV file is rather confusing, I often asked myself if we were referring to the restore file or the CSV. Rewording my help here (applies to UI and CLI).
- Strings in the lang file should ideally be alphabetically ordered
- It might be safer for later use to place the defaults values in the form in their respective array instead of mixing them with the options: $formdata['defaults']['foo'] = 'bar'; And if so, perhaps the function tool_uploadcourse_process_course_upload() could get the defaults as an argument.
- CC* variables and constants should be renamed to match the frankestyle name of this tool.
- The form should use setType() on 'ccshortname' and 'contextid'
- In tool_uploadcourse_process_template(), the letters flu do not match sfi in the callback function tool_uploadcourse_process_template_callback().
I am willing to take over the work to be done from here, but we first need to decide whether we want to change the structure of this code or not?
Thanks everyone for your involvement in this issue!
|Peer review in progress [ 10013 ]
|Development in progress [ 3 ]
Hi Fred -
Thanks for taking a look at this for me - it's much appreciated. I think all that you have pointed out, and your outlined approach for dealing with it is fine apart from the following:
- I think there should be an option to select a course as template should remain - if the length of the list is the problem then a user should just type in a valid shortname
- the oldshortname renaming process is modelled on the user uploader admin tool - infact the entire flavour and coding style is modelled on this deliberately as it seems like a supported core functional approach. However - the extended procedural nature of the code is now an obstacle, so it does need proper encapsulation.
- the use of multiple copyrights was done to indicate where the ideas came from to create the code - irrespective of how little/great the contribution. If we were to strip it back down to who actually created the files then that would be me.
All in all - I would be very happy if you were to take it forward from here - the main thing is that this functionality is long overdue in core Moodle, so if you can do that I'll be cheering you on.
Piers Harding.
Thanks Frédéric - great to hear you're able to pick this up and work on it (re-assigning this to you) - is this something you can finish for inclusion in 2.6?
btw - when Piers first did this work Unit Tests weren't happening much in core and there wasn't a lot of point adding them until someone had managed to review the code and provide feedback on structure etc - great that you have managed to do this - thanks!
|Dan Marsden [ danmarsden ]
|Frédéric Massart [ fred ]
|Ranked lower
|Ranked higher
How are you going Frédéric?
And @Michael: what are the implications of taking this out of Sprint 1? Slight lowering of priority? Progress too slow?
Hi Derek,
I have been actively working on this issue, but I need to rewrite quite a lot of it to allow for proper Unit Testing. It has not really been taken out of the sprint, it has been pushed in the Sprint 2 as I could not finish it in the previous sprint. I am hoping to push it for peer review in the next couple of days.
|Story Points
Hi Fred,
Just skim through at the end of the day with a big disclaimer that i've not fully considered it at all, but to write it down whilst its fresh in my head:
- The handling of the numeric shortname caught my attention
- I think we allow all numeric shortnames, so it could be problematic, if you are assuming that is a template placeholder.
- In a few places you are doing is_numeric checks for a placeholder. I wondered if you could instead do the conversion to a template on 'input'
- In the processor, rather than using in_array() with a list of options, I think that you could make the code easier to understand/modify with a switch statement or default (but that might be my personal preference)
- get_final_update_data/get_final_create_data phpdoc needs to clarify difference between them
- The OUTCOME_ xxx names are a bit confusing with moodle terminonology of outcomes being related to a course. I would suggest considering something different
|BACKEND Sprint 2 [ 4 ]
|patch triaged
|docs_required patch qa_test_required triaged
|Testing Instructions
# Create a CSV to import data from
# Navigate to Home / Site administration / Courses / Upload courses
# *TEST*: Different combinations of settings
# *TEST*: Different combinations of CSV data
# *Make sure*
#- Expected errors occurred
#- Update/Create/Delete/Rename, etc... worked when no errors occurred
|Pull Master Diff URL
|Pull Master Branch
|Pull from Repository
Here is my first patch pushed for peer review.
Dan, the shortname can be 0 according to course/edit.php, and so I implemented that check. Also, I do not want to clean the data as it could potentially lead to random behaviours. I also improved the readability following your comments.
For further reference, at this stage, the CSV accepts the following data:
Course data
|Category ID
|Each node separated by ' / '
|Passed to strtotime()
I intentionally removed numsections and other course format related options because they should be handled properly for any course format possible.
Required fields:
- shortname (unless shortname template is set and mode does not allow for updates)
- fullname (on create)
- category (on create)
Enrolment data
|The enrolment 1 method
|The value for any option for enrolment 1
|Disables the enrolment method
|Deletes the enrolment method
|5 days
|Timestamp or passed to strtotime()
|1st July 2012
|Passed to strtotime()
|1st July 2012
|Passed to strtotime()
|Role shortname
Role renaming
Use role_shortname.
|Super Teacher
|Great Student
Extra options
|Shortname to rename the course with
|Reset the course after import
|Delete the course before any other update
|Backup file of a course to restore over the created/updated course
|Shortname of a course to restore over the created/updated course
There are some possible cache improvement, but I believe they could be part of subtask of this issue once we have a solid base. Same for the course format options, they could be delayed to another issue.
Not sure what more information I can give so far, please ping me for any more information on how and why it works.
Edit: Removed summary from the required fields.
|Development in progress [ 3 ]
|Waiting for peer review [ 10012 ]
|Waiting for peer review [ 10012 ]
|Peer review in progress [ 10013 ]
|Peer reviewer
Hi Fred,
Great work, I really like the look of this!
Some (mostly small) suggestions:
1. You don't need to do $CFG->admin, it'll be auto-translated
2. I think you've got some leftover includes in there (at least, I can't see why you need clilib there)
3. I dislike the variable/param name iid, how about importid? "Variable names should always be easy-to-read, meaningful lower-case English words."
4. In the interface, i'm not especially convinced by the modes - I think it would be good to ask Mary Cooch if she can try it out, and she might able to help us come up with a simple way to explain it. I'm wondering if it should be something much more explicitly like "How to handle new courses". Its tricky - hoping Mary can help!
5. I continue to think you could split up prepare() into more manageable chunks, but I suppose you disagree
6. Line 642 looks unfinished "// We cannot"
7. Line 248: Phpdocs for function tool_uploadcourse_course::exists has incomplete parameters list
8. Line 835: Phpdocs for function tool_uploadcourse_course::reset has incomplete parameters list
9. Well firstly, I always think that using a static class property is slightly strange. Doesn't seem like there is need for it to be visible across the whole instance at the moment. But really quite soon you're gonna run into probably and need to add a reset method for unit testing, which would then make it make sense - but..
10. Ideally you'd be using an add-hoc MUC cache and you'd get reseting for unit tests for free here. Its partly why we added the MUC framework, to ensure everything is using consistent reset. The code is trivial:
$cache = cache::make_from_params(cache_store::MODE_SESSION, 'repository_skydrive', 'foldernamelist');
$cache->set('key', $value);
11. Line 30: Class tool_uploadcourse_helper does not have @copyright tag
12. Line 30: Class tool_uploadcourse_helper does not have @license tag
13. Line 260: Phpdocs for function tool_uploadcourse_processor::get_restore_content_dir has incomplete parameters list
With the tests..
15. When I run the whole of phpunit tests I got (maybe related to other reseting problems I mentioned)
There was 1 error:
1) tool_uploadcourse_helper_testcase::test_get_restore_content_dir
backup_helper_exception: Backup is missing XML file: /Users/danp/moodles/pm/moodledata_phpu/temp/backup/86729a23efa6e5b12e37451c68a4c66d/moodle_backup.xml
To re-run:
vendor/bin/phpunit tool_uploadcourse_helper_testcase admin/tool/uploadcourse/tests/helper_test.php
Tests: 1967, Assertions: 36191, Errors: 1, Skipped: 5.
16. Its great to see all this really well unit testable code. Can I suggest you run the code coverage report on the tests and try and get better coverage of some of the bits that are missing at the moment. Looks like this plugin can be a great example that we can show people of the sort of testing we want on new plugins
17. For bonus points, you could add a few behat tests.
18. There are a few trivial issues picked up by the code checker, it would be good to fix them. (Again to make this a shining example plugin )
Once again, great work!
|Peer review in progress [ 10013 ]
|Development in progress [ 3 ]
|docs_required patch qa_test_required triaged
|docs_required patch qa_test_required triaged ui_change
Thanks a lot for your review Dan.
I have taken car of all the comments you made above. The unit testing coverage has been improved though it does not answer 100% from PHP_CodeCoverage, but this is pretty minor. I have also added 2 basic Behat tests for bonus points, and MUC for caching. You were right, the failed tests were caused by the use of class variables.
Would you mind having a second look at the Processor/Tracker classes as those are the ones I'm more concerned about in terms of design.
I forgot to mention that I agree on the wording problem, but I think that would be easier to fix up once this is integrated, so that it can be reviewed in place by Mary, Helen, etc....
|Development in progress [ 3 ]
|Waiting for peer review [ 10012 ]
Pushing this for integration after chatting a bit with Dan about it.
Further improvements that we should keep in mind:
- Handling of course format options
- Handling of course summary files
- Caching when checking if a course exists (tricky as it needs proper very accurate invalidation)
|Waiting for peer review [ 10012 ]
|Waiting for integration review [ 10010 ]
+1, sorry I forgot to press the button myself. I've played with it quite a bit and really like it
I'm sorry to have to ask a newbie question: how do I help test? I can set up a test server, but how do I get the code and/or patch? Thanks!
Hi Elizabeth,
Thanks for volunteering! If you are familiar with checking out a git branch, you can use fred's branch: git://github.com/FMCorz/moodle.git
Otherwise, if you can wait a week, this will hopefully get integrated and should be available for testing on qa.moodle.net.
|Currently in integration
|Yes [ 10041 ]
|Waiting for integration review [ 10010 ]
|Integration review in progress [ 10004 ]
Hi guys. This is really a huge work and very nice code!
Fred, just to notice couple of things that don't break anything but may be worth looking at:
- get_plugin_list() is deprecated, also does not make sense to cache it's result in MUC any more, it will only make it slower.
- cache 'helper' should be persistent (I tried setting it to persistent and reduced unittest time by 30%). But you need to set some value for 'persistent_max_size' then to make it optimal. Why not have two cache definitions, one for lists of plugins/roles and another, with maximum size, - for categories and backup ids? Or maybe even three. Caches need good settings. 'simplekeys' can be used for anything except categories paths.
Fred, also the plugin name should be included in standard_plugins_list()
Thanks Marina,
I have updated my patch to add the plugin to the standard ones, and I removed the use of the get_plugin_list() (and its local cache). In regard with the cache, I am not sure why there is such an improvement when executing the Unit Tests, the default cache for request is a static store, which does pretty much the same than the persistent cache.
At first I was not going to implement MUC in that issue, but at the end I did to make it easier to invalidate the cache during Unit Tests. I don't mind improving the cache of this plugin but I don't think this is very important. The purpose of the caching here was only to prevent fetching twice the same result, it's not like the logic is used on every page of Moodle... it is only when uploading a CSV, which is already a heavy process that some caching takes place.
Sure I could create new cache definitions, to use persistent and simpley keys. But using persistent with a request cache seems counter productive to me... the default store being a static store, and the logic to fetch the information from there being quite straightforward, what is the real deal? Well, maybe the key doesn't need to be hashed any more, but really, 30% of improvement?
Also, that makes me think that we should actually never use persistent with a request cache. We end up caching things twice, for no apparent reason. And if I have my own specific store for request caching, then it will never be used because most of the data will be cached statically, which defeats the purpose again.
Could we eventually delay this caching improvement to another issue, which could also handle the caching/invalidating of the exists methods from course which could be called a lot of time when incrementing shortnames?
Based on discussion with Sam, I've added one more commit to cache the cache loader. The persistent option also creates a singleton of the cache loader and returns it whenever the cache loader is requested again. To benefit from this improvement, but not using the persistent data which does not make much sense on request loaders (see comment above), I am now caching the cache loader instance into a static variable. It seems like Unit Tests are faster using this method.
Fred, I feel that templatecourse and backupcourse are the same. In the logic of my own code I simply do:
if a course-template file is given in the file or a course to copy from, the same procedure is started for "new" and "copy" courseaction. The existing course is backed up on the fly and restored to get a copy of it which becomes the new course.
I still appreciate the big work having been done so far. Rosario
Hi Rosario, I am not sure to follow what you mean. Here are the possible options:
- You upload a course backup to restore on top of the created courses.
- You select a course which will be backed up on the fly and restored on the created courses.
- In the CSV, for a specific course, you specify the path to a backup file.
- In the CSV, for a specific course, you specify the shortname of a course to create a backup from.
Some options take precedence on others. So yes, you are right, those options are resulting in the same operation, but if you are offered to chose to restore from an existing course, or a backup file.
I am sorry, I was not aware that you include uploading a whole backup-file. But this is for certain another fine option.
Dan, I'm not as familiar as I need to be with git yet-- if I check out the master branch you indicated, does that get all of Moodle Core, or do I need to do that first? Feel free to point me toward "Moodle Git for Dummies" as appropriate. I do want to learn to do this so I can help with testing.
|Testing Instructions
# Create a CSV to import data from
# Navigate to Home / Site administration / Courses / Upload courses
# *TEST*: Different combinations of settings
# *TEST*: Different combinations of CSV data
# *Make sure*
#- Expected errors occurred
#- Update/Create/Delete/Rename, etc... worked when no errors occurred
# Create a CSV to import data from
# Navigate to Home / Site administration / Courses / Upload courses
# *TEST*: Different combinations of settings
# *TEST*: Different combinations of CSV data
# *Make sure*
#- Expected errors occurred
#- Update/Create/Delete/Rename, etc... worked when no errors occurred
Repeat using the command line tool.
Thanks everybody participating in the project, it has been integrated in master.
The cache loader caching commit was not integrated, Sam has promised to raise a separate issue about request cache.
|Integration review in progress [ 10004 ]
|Waiting for testing [ 10005 ]
|Fix Version/s
|2.6 [ 12579 ]
|Fix Version/s
|DEV backlog [ 10464 ]
|Fix Version/s
|Testing Instructions
# Create a CSV to import data from
# Navigate to Home / Site administration / Courses / Upload courses
# *TEST*: Different combinations of settings
# *TEST*: Different combinations of CSV data
# *Make sure*
#- Expected errors occurred
#- Update/Create/Delete/Rename, etc... worked when no errors occurred
Repeat using the command line tool.
# Create a CSV to import data from
# Navigate to Home / Site administration / Courses / Upload courses
# *TEST*: Different combinations of settings
# *TEST*: Different combinations of CSV data (https://tracker.moodle.org/browse/MDL-13114?focusedCommentId=231131&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-231131)
# *Make sure*
#- Expected errors occurred
#- Update/Create/Delete/Rename, etc... worked when no errors occurred
Repeat using the command line tool.
Is this going to be integrated into the next weekly build for 2.4.4 or is it just specific to 2.6?
Susan - stable releases like 2.4.x/2.5.x are only supported for bug fixes - this is a new feature so will only land in 2.6
Susan, also it uses some 2.6-specific features like autoloading so unfortunately this plugin can't be installed on previous versions without modifications
No worries! I just wasn't sure. Thank you for letting me know. I'm happy to continue using Pier's script manually until we get to 2.6.
but of course... Piers original tool (which this feature is based on) should work on 2.5 if you need something in the interim:
- cli help says "--action=createnew", should it be "mode"?
Not real bugs:
1/ frontpage shortname collision always results in error - is that expected?
2/ allow deletes,resets and allow renames are not documented, I guess help icon would help
3/ why show shortname template if it is present in the csv file?
4/ if I have collision in shortname but unique idnumber the template is not used - I am confused
5/ where is the list of allowed CSV fields?
6/ help says shortname is required, it is not - you can use template in web UI
7/ for all update operations I would expect only idnumber or shortname + one other field to be required
I did not find any real bugs, so feel free to create new issues (for usability improvements and improved docs) and mark this as tested.
|Testing in progress [ 10011 ]
|Problem during testing [ 10007 ]
Hi Petr,
thanks for the thorough test!
- I've added a commit to fix the typo
1/ I am not sure what you mean here, I have been able to update my frontpage using the tool. That actually makes me think that should probably be prohibited, what do you think? If yes, it would be better to take care of it in another issue.
2/ I have raised
MDL-40728 to improve the UI documentation
3/ This can be a bit confusing, the UI defines the defaults, where as the CSV defines a property for that particular course. Though, you cannot specify a shortname template (string to generate a shortname from) in the CSV itself, because it doesn't make sense as you already have the information on the same line... If you're talking about the template course (course to import content from after upload), this can be define in both the UI and the CSV.
4/ The template is only used if it is safe to do so. The only occasion it would be used is when the user would expect it, by leaving the shortname field empty. I tried not to randomise the behaviours too much here. If you want to use a shortname template, you need to create new courses, and not provide any shortname.
5/ For now, nowhere.
MDL-40728 could add them in the UI, but Moodle Docs is probably the best place (nothing there yet).
6/ You are right, this is wrong, where about have you seen that?
7/ Only shortname is supposed to be required for that operation, ID number is never used as a course identifier. Could you describe how to replicate this? What other fields were mentioned as required?
Many thanks!
|Problem during testing [ 10007 ]
|Waiting for testing [ 10005 ]
Passing the test, the other cosmetic issues will be addressed separately.
|Testing in progress [ 10011 ]
|Tested [ 10006 ]
1/ At first I accidentally added course with the same shortname as the frontpage, so I tried to select update or increment and after hitting the Preview button there was still the red X
6/ there is only one help - it is the icon next to the Page title on the first form
7/ For me idnumber is the external course identifier, shortname is internal identifier. So I would expect to be able to use any of them as unique identifier, their only difference is that idnumber may be empty.
Thanks Petr, I have raised
MDL-40737 to prevent any alteration of the front page. Thanks for spotting that! About the ID number, the original plugin didn't allow this feature, so I have to admit that it didn't occur to me. While I agree with you, I think we should leave it the way it is for now and wait for a feature request to allow the selection by ID number.
Moodle has many old functions,
And although they cause no malfunction,
There comes a day,
When they get deprecated away,
And get and put on the list for expulsion.
Thanks for all the reports/testing/fixes this week. This issue has been sent upstream.
|Tested [ 10006 ]
|Closed [ 6 ]
|Fixed [ 1 ]
|Currently in integration
|Yes [ 10041 ]
|Integration date
Many thanks to [Piers Harding, who originally created the plugin, and anyone who has been involved.
As Dan Marsden mentioned above, for users of Moodle below 2.6, the original plugin is still available here: https://github.com/piersharding/moodle-tool_uploadcourse.
Hey - a big thank you Frédéric for all your hard work on getting this through and also to Dan, Petr, Marina, Damyon (especially for the poetry) and many more. I can now retire my plugin!
Piers Harding.
First off thanks to everyone for such a great new feature. A quick question - what is the recommended way to uniquely identify courses for deletion? I was expecting to be able to use the 'id' of the course but it looks not possible. If I use shortname and idnumber will it combine them? Is there an order of precedence? Cheers
Hi Alastair, the plugin only uses the shortname as an identifier. Even if the ID number was used to identify the courses, you would not need to combine it with a shortname, because both will always be unique. The plugin also has a "Preview" mode, which would give you a quick overview of what actions would be undertaken when you will proceed with the CSV, so you should not end up with unexpected deletions. Cheers!
Thanks Frédéric
Unfortunately as shortname is not constrained to be unique at a schema level (or even non-empty) we have a lot of courses that have been automatically created with no shortname, or with duplicate shortnames. Many of these do however have an ID number but I don't think this can be guaranteed to be unique either. Do you have any advice on how to tackle this? At the moment I think I will just have to modify the code to use the course ID as an identifier instead of the shortname as this seems be the only reliably unique and existent identifier.
Moodle short course names must be present and must be unique, if you somehow inserted them directly into the database it is your responsibility to fix them I am afraid.
I am surprised you didn't encounter any issues as Moodle really assumes that shortnames and ID numbers are unique... Indeed I would advise you to edit the plugin code to use IDs, though the best is probably to set proper shortnames in each of your courses.
Thanks for the advice - I have decided to copy the course id into the shortname where it is empty, and append it to the shortname where it is duplicated. For reference:
UPDATE mdl_course SET shortname = id WHERE shortname = '';
UPDATE mdl_course
SET shortname = CONCAT(shortname,'-',id)
WHERE shortname IN
SELECT shortname FROM
SELECT shortname
FROM mdl_course
GROUP BY shortname HAVING count(id) > 1
AS shortnames
I have a Moodle course that contains a comma in the shortname - it seems this is permitted by the Moodle interface for course creation. This obviously yields the following error from this tool:
"There is something wrong with the format of the CSV file - please check the number of headings and columns match, and that the delimiter and file encoding are correct (don\t use comma-quoted as Moodle does not support it): Invalid CSV file format - number of columns is not constant!"
This suggests that I may not escape the comma by simply enclosing it in quotes. Is there any way around this?
I bet the problem is that your CSV is not a proper CSV. Try to open it in LibreOffice Calc or Excel and save it again. There should not be any issues with quoted fields, if there were the fields like "Summary" would be a real problem! We should actually update this message: "(don\t use comma-quoted as Moodle does not support it)" as it is incorrect.
Thanks for your reply Frédéric
The CSV is exported from MySQL Workbench, I have pared it down to the offending line (that causes the aforementioned error):
According to RFC4180/2.6, "Fields containing line breaks (CRLF), double quotes, and commas should be enclosed in double-quotes." it seems as though the comma has been escaped in accordance with at least this particular definition of a proper CSV - would you agree that that is the correct method of escaping a comma?
Thanks again
Yes, I agree, and it works. I have just copied/pasted your small example into a new CSV file and I could preview the CSV without any issue. Have you selected the right encoding? Mine was UTF-8.
Still no dice unfortunately - I am running this tool under Moodle 2.2.6 (Build: 20121112) however, do you think that might be the issue? Would you mind attaching the CSV you just created so I can try to rule out any encoding issues etc?
I think I'll just use tab delimited files as a workaround for now.
Thanks for your help
I have had some limited success with using the tab delimited format. My issue now is that any quoted fields are unable to be located, e.g.:
Course not deleted - missing 6 "Staff Area Old" 1
Moodle doesn't seem to strip the quotes off so the shortname doesn't match - is this likely an issue with Moodle 2.2.6? Do quoted shortnames work for anyone else? I have a 2.5 install I can try
Having tested in 2.5.0 it seems it was an issue with 2.2.6
My solution is to use un-quoted, tab delimited data and 2.2.6 seems happy with that.
Sorry to clutter these comments with my progress, hopefully it may be of some use to others.
Hi Alastair. I'm glad you mentioned that you are using Moodle 2.2 because we have fixed the CSV comma/quote issues in 2.4, see
Apologies for the confusion, we are using this in 2.2 to perform some course housekeeping prior to an upgrade to 2.5 - now working very nicely indeed with tab delimited data
It would be great if cohorts could be added to uploaded courses. Any chance on that? I've been using the third party upload course plug in and have missed this feature.
QA test added here for inclusion in 2.6 testing MDLQA-5738 Please note: I would be very grateful if someone would check the test out, as the course create process is very new to me
Removing the label docs_required. http://docs.moodle.org/26/en/Upload_courses.
Arg... I just realised that Mary Cooch had already done some work there... I'll try to merge those!
|docs_required patch qa_test_required triaged ui_change
|patch qa_test_required triaged ui_change
|patch qa_test_required triaged ui_change
|patch triaged ui_change