This is a very old thread, but the link posted above to the calibration and homing documentation still has information about the process.
Additionally, you can try checking your wiring or changing your settings as mentioned in these encoders troubleshooting tips.
If you search the forum for “encoders” and sort by date, you should find some more recent posts. If you don’t find an answer to your issues, please feel free to start a new topic.